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Abstract 


This  research  uses  Decision  Analysis  (DA)  to  develop  a  structured,  repeatable, 
and  most  importantly,  defensible  decision  model  for  the  evaluation  of  proposed  IED 
defeat  solutions  submitted  to  the  Joint  IED  Defeat  Organization  (JIEDDO).  Additive 
value  models  using  Value -Focused  Thinking  (VFT)  and  the  Analytic  Hierarchy  Process 
(AHP)  are  examined  as  possible  methodologies.  VFT  is  determined  to  be  the  “best  fit” 
for  JIEDDO’ s  decision  situation  in  which  proposals  are  submitted  continuously  and  must 
be  scored  independently  of  previous  proposals.  VFT  is  first  used  to  detennine  desirable 
qualities  in  IED  defeat  options,  and  then  to  generate  a  hierarchical  value  model  to 
evaluate  these  qualities  in  a  selected  group  of  alternatives.  The  most  important  criteria 
for  IED  defeat  proposal  evaluation  are:  Need  for  the  Capability,  Operational 
Perfonnance,  and  Usability.  A  group  of  30  proposals,  previously  assessed  by  the  current 
JIEDDO  process,  is  evaluated  using  the  VFT  decision  model  and  the  rank  ordered  results 
are  compared  with  JIEDDO ’s  previous  selection  decisions.  The  VFT  decision  model 
results  support  JIEDDO ’s  past  decisions  to  accept  or  reject  IED  defeat  proposals, 
validating  the  model.  Sensitivity  analysis  is  then  conducted  to  allow  further  insight  to  the 
robustness  of  the  model.  The  resulting  effort  creates  a  decision  support  model  that,  when 
consistently  applied,  provides  repeatable  and  defensible  decisions  that  reflect  JIEDDO ’s 
priorities  for  proposal  selection. 
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DEVELOPING  A  DECISION  MODEL  FOR  JOINT  IMPROVISED  EXPLOSIVE 


DEVICE  DEFEAT  ORGANIZATION  (JIEDDO)  PROPOSAL  SELECTION 


I.  Introduction 


JIEDDO 

In  recent  years,  improvised  explosive  devices  (IEDs)  have  emerged  as  the  most 
significant  threat  to  U.S.  forces  deployed  in  Iraq  and  Afghanistan.  IEDs  are  the  primary 
source  of  coalition  casualties  in  these  countries,  accounting  for  about  2/3  of  all  combat 
related  deaths  in  Iraq  since  2003,  and  about  1/2  of  combat  related  deaths  in  Afghanistan 
since  2001  (Atkinson,  2007).  IEDs  are  developed,  emplaced  and  detonated  by  complex 
and  adaptable  insurgent  networks  with  extensive  practice  dating  back  to  the  Russian 
occupation  of  Afghanistan  and  the  Iran-Iraq  War  (Global  Security  Organization,  2007). 

In  addition,  the  number  of  IEDs  placed  by  enemy  forces,  either  exploded  or  discovered 
prior  to  detonation,  increased  six  fold  between  2003  and  2006  (JIEDDO,  2006).  IEDs  are 
the  insurgent  weapon  of  choice:  They  are  easy  to  make,  low  cost,  difficult  to  defend 
against,  and  cause  untold  death  and  destruction  (Chertoff,  2007). 

In  October  2003,  the  Army  Chief  of  Staff  established  the  Anny  IED  Task  Force 
to  orchestrate  efforts  to  eliminate  the  IED  threat  to  ground  forces  in  Iraq  and  Afghanistan. 
The  goal  of  the  task  force  was  to  recommend  the  best  available  responses  and  develop 
and  field  the  selected  solutions  for  IED  countermeasures.  Although  the  Army  was  the 
primary  force  dealing  with  IEDs  in  theater,  it  was  necessary  to  synchronize  the 
Department  of  Defense’s  (DoD)  response  to  the  IED  threat.  In  July  2004,  the  task  force 
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was  transformed  into  the  Joint  IED  Defeat  Task  Force.  The  following  year,  by  DoD 
Directive  2000. 19,  the  command  structure  was  modified  so  that  the  Director  of  the  Joint 
IED  Defeat  Task  Force  reported  directly  to  the  Deputy  Secretary  of  Defense  and  the 
process  for  fielding  life-saving  technologies  was  further  streamlined.  In  December  2006, 
the  Joint  IED  Defeat  Organization  (JIEDDO)  was  established  as  a  permanently  manned 
entity.  The  creation  of  JIEDDO  was  intended  to  orchestrate  all  available  resources  within 
the  DoD,  the  private  sector,  our  allies  and  our  coalition  partners  to  combat  the  IED  threat 
(JIEDDO,  2007). 

JIEDDO ’s  mission  is  to  focus  (lead,  advocate,  coordinate)  all  Department  of 
Defense  actions  in  support  of  Combatant  Commanders'  and  their  respective  Joint  Task 
Forces'  efforts  to  defeat  Improvised  Explosive  Devices  as  weapons  of  strategic  influence 
(Department  of  Defense,  2006).  JIEDDO’s  objectives  are  to: 

•  Rapidly  and  systematically  reduce  the  effect  of  all  forms  of  IEDs  against  Joint 
and  Coalition  forces  ahead  of  enemy  innovation. 

•  Provide  senior  leaders  a  single  point  of  contact  for  counter-IED  efforts 
throughout  the  Department  of  Defense. 

•  Establish  a  Joint  Common  Operational  Picture  (JCOP)  of  IEDs  and  their 
employment  in  the  Global  War  on  Terror. 

•  Provide  a  Joint  forum  to  collect  and  synchronize  all  applicable  efforts  of  the 
Department  of  Defense  and  other  agencies. 

•  Provide  senior  leaders  a  rapid  and  accountable  method  for  identifying  issues 
requiring  interservice  resourcing  and  prioritizing  decisions. 
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•  Provide  a  Joint  Forum  to  identify  counter-IED  efforts  and  requirements  to  be 
rapidly  implemented,  accelerated  or  developed  in  the  long  term. 

•  Provide  a  forum  of  interservice,  interagency,  industry  and  international 
coordination  of  IED  defeat  (JIEDDO,  2007). 

Improvised  Explosive  Devices  (IEDs) 

IEDs  comprise  a  wide  variety  of  homemade  devices  that  are  designed  to  cause 
death  or  injury  by  using  explosives  alone  or  in  combination  with  toxic  chemicals, 
biological  toxins,  or  radiological  material.  IEDs  can  be  produced  in  varying  sizes, 
functioning  methods,  containers,  and  delivery  methods.  IEDs  can  use  commercial  or 
military  explosives,  homemade  explosives,  or  military  ordnance  and  ordnance 
components.  They  are  unique  in  nature  because  the  IED  builder  has  to  improvise  with 
the  materials  at  hand.  Designed  to  defeat  a  specific  target  or  type  of  target,  they  generally 
become  more  difficult  to  detect  and  protect  against  as  they  become  more  sophisticated 
(Global  Security  Organization,  2007). 

IEDs  fall  into  three  types  of  categories:  Package  Type,  Vehicle-Borne,  and 
Suicide  Bomb.  Though  they  can  vary  widely  in  shape  and  form,  they  share  a  common  set 
of  components  and  consist  of  an  initiation  system  or  fuse,  explosive  fill,  a  detonator  and 
power  supply,  and  a  container.  IEDs  are  relatively  simple  devices,  but  they  are  extremely 
effective  as  an  asymmetric  strategy  for  insurgents  (Global  Security  Organization,  2007). 
Current  IED  Defeat  Practices 

Defeating  IEDs  requires  a  comprehensive  approach.  In  the  interest  of  this, 
JIEDDO ’s  current  counter-IED  (C-IED)  practices  are  arranged  into  three  separate  lines  of 
operation  (LOOs):  Attack  the  Network,  Defeat  the  Device  and  Train  the  Force. 
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•  Attack  the  Network  is  the  offensive  portion  of  C-IED  efforts  intended  to 
interdict  the  network  responsible  for  the  production  and  emplacement  of  IEDs. 

•  Defeat  the  Device  describes  efforts  to  counter  the  impact  of  IEDs  after  they 
have  been  successfully  emplaced  by  the  enemy. 

•  Train  the  Force:  JIEDDO  supports  individual  and  collective  training  programs 
to  prepare  units  prior  to  and  during  deployments. 

These  LOOs  are  further  subdivided  into  five  tenets  of  C-IED  warfare: 
Prevent/Predict  (Attack  the  Network),  Detect,  Neutralize,  Mitigate  (Defeat  the  Device) 
and  Train  (Train  the  Force).  When  new  proposals  for  C-IED  technology  are  submitted  to 
JIEDDO,  they  are  assigned  to  one  of  the  five  tenets  within  the  LOO  structure  when  being 
considered  for  funding  (JIEDDO,  2007). 

JIEDDO  is  currently  in  the  process  of  adopting  a  revised  set  of  six  tenets  (Bonder, 
2008).  These  new  tenets  will  be  used  for  the  purposes  of  this  research  paper  in  the 
evaluation  of  the  proposal  set. 

•  Predict:  Activities  to  anticipate  threat  actions  by  integrating  intelligence  to 
provide  a  clearer  understanding  of  enemy  personnel,  equipment, 
infrastructure,  processes  and  actions. 

•  Prevent:  Activities  to  target,  interdict,  and  destroy  key  enemy  personnel, 
infrastructure,  logistics  capabilities,  and  enemy  combat  operations  involving 
IEDs  prior  to  emplacement. 

•  Detect:  Activities  to  identify  and  locate  enemy  personnel,  explosive  devices 
and  their  component  parts,  equipment,  logistics  operations  and  infrastructure 
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through  the  act  of  emplacement  in  order  to  provide  accurate  and  timely 
information  to  military  operations  and  planners. 

•  Degrade:  Activities  to  disrupt,  deceive,  and  avoid  the  IED  chain  of  events  and 
reduce  public  support,  and  to  ideally  deter  the  enemy  use  of  IEDs  by 
increasing  the  cost  (time,  dollars,  personnel,  and  risk)  of  employing  IEDs. 

•  Neutralize:  Activities  to  eliminate  the  hazards  of  enemy  IEDs  once  emplaced 
by  destroying  them  or  rendering  them  incapable  of  detonating  at  the 
time/place  of  the  enemy’s  choosing. 

•  Protect:  Activities  to  protect  personnel  and  equipment  by  disrupting, 
channeling,  blocking  or  redirecting  blast  energy  or  fragmentation.  (Bonder, 
2008) 

The  nature  of  the  IED  threat  requires  rapid  selection,  funding  and  fielding  of 
C-IED  systems.  However,  JIEDDO  faces  the  same  challenges  as  all  DoD  services  in 
following  federal  acquisition  rules.  JIEDDO ’s  acquisitions  are  governed  by  the  Joint 
IED  Defeat  Capability  Approval  and  Acquisition  Management  Process  (JCAAMP). 
JCAAMP  establishes  policy  for  evaluating  C-IED  proposals  sponsored  by  JIEDDO  and 
assigns  responsibilities  for  these  evaluations  (JIEDDO,  2007).  The  intent  of  JCAAMP  is 
to  rapidly  assess  and  acquire  new  C-IED  capabilities  and  quickly  provide  them  to  the 
warfighter.  The  Military  Services,  the  Joint  Staff,  OSD  and  the  Combatant  Commanders, 
along  with  teams  of  scientists  and  other  subject  matter  experts,  provide  validation  and 
oversight  throughout  this  process  (JIEDDO,  2006). 

The  selection  of  a  methodology  for  JIEDDO ’s  decision  problem  requires  an 
understanding  of  the  current  decision  process  and  where  the  decision  model  is  intended  to 
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be  used  in  this  process.  JIEDDO  Instruction  5000.0 1 ,  Joint  IED  Defeat  Capability 
Approval  and  Acquisition  Managemen  t  Process  (JCAAMP),  is  the  guidebook  for  the 
current  proposal  selection  process.  This  Instruction  defines  and  implements  the  rapid 
acquisition  management  responsibilities  for  counter-IED  efforts  (JIEDDO,  2007). 

The  existing  process  for  the  selection  of  individual  proposals  begins  with  the 
issuance  of  a  Broad  Area  Announcement  (BAA)  via  the  JIEDDO  website.  These 
announcements  generate  proposals  from  industry,  academia  and  research  and 
development  institutions.  In  addition,  unsolicited  proposals  may  also  be  received  from 
these  sources.  These  incoming  proposals  are  accepted  into  JIEDDO’s  BAA  Information 
Delivery  System  (BIDS),  which  aids  in  the  multilayered  evaluation  process  prior  to  being 
presented  to  senior  leadership  for  a  decision  to  accept  or  reject  the  proposal. 

The  BIDS  process  separates  incoming  proposals  into  two  groups:  those  that  have 
the  potential  to  perfonn  the  IED  Defeat  mission,  and  those  concepts  that  are  not  feasible. 
The  use  of  such  a  filter  is  especially  critical  in  view  of  the  number  of  proposals  received 
by  JIEDDO— 1274  in  2006  and  2007  alone.  Of  these,  only  447  passed  the  initial  review 
(Jacoby,  2008).  Therefore,  the  focus  of  this  research  is  to  create  a  decision  model  to 
support  the  initial  decision  to  accept  or  reject  the  proposal.  The  objective  is  to  provide  a 
defensible  and  repeatable  framework  that  will  give  added  justification  and  credibility  to 
the  decision  to  push  a  proposal  forward  in  the  evaluation  process. 

The  proposals  selected  through  BIDS  enter  the  JCAAMP  process  at  transition 
point  1  (TP1),  depicted  in  Figure  1.  Proposals  that  make  it  past  TP1  are  endorsed  by  the 
Deputy  Director  of  JIEDDO  as  meeting  a  current  need  and  being  mature  enough  to  enter 
the  rapid  acquisition  and  development  process  (JIEDDO,  2007). 
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-Initiative  Evaluation  Process- 


Figure  1.  The  JCAAMP  Process  (JIEDDO,  2007) 

Research  Objective 

JIEDDO  receives  numerous  proposals  from  various  sources  aimed  at  reducing  the 
threat  posed  by  IEDs.  The  importance  of  focusing  the  C-IED  effort  on  the  right  mix  of 
functional  areas  requires  that  JIEDDO  choose  proposals  capable  of  providing  the  most 
effective  portfolio  of  C-IED  capabilities  to  the  warfighter  within  budget  constraints. 

Currently  no  Decision  Analysis  model  is  used  to  provide  traceable,  repeatable  and 
defensible  scoring  of  competing  proposals  submitted  to  JIEDDO  for  consideration.  A 
model  of  this  type  would  use  evaluation  measures  based  on  JIEDDO ’s  stated  operational 
objectives  to  accurately  represent  the  value  that  decision  makers  place  on  individual 
objectives.  Use  of  such  a  model  will  enable  the  JIEDDO  to  optimize  the  proposal 
selection  process  and  ultimately  decrease  the  impact  of  the  IED  threat. 

The  objective  of  this  effort  was  to  develop  a  repeatable  framework  to  effectively 
assist  JIEDDO  in  selecting  proposals  for  funding  that  would  provide  the  most  effective 
C-IED  capabilities  within  budget  constraints.  Using  the  Decision  Analysis  approach  of 
Value-Focused  Thinking  to  elicit  the  desired  objectives  of  JIEDDO  decision  makers,  a 
value  hierarchy  was  built.  Sensitivity  analysis  was  conducted  to  identify  areas  for  future 
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refinement  and  modification  of  this  model  according  to  guidance  provided  by  JIEDDO. 
The  finished  product  offers  useful  insight  to  JIEDDO  leadership  and  ensures  that 
proposal  acceptance  is  based  on  the  value  hierarchy  developed  by  the  AFIT  research 
team  and  approved  by  JIEDDO.  The  approved  model  is  intended  to  be  used 
independently  by  JIEDDO  to  evaluate  incoming  proposals. 

Research  Scope 

JIEDDO  must  consider  a  widely  varied  and  constantly  evolving  threat  when 
selecting  proposals  for  funding,  so  the  analysis  of  submitted  proposals  is  a  complex 
endeavor.  Since  funds  are  not  strictly  divided  among  the  established  LOOs  (Attack  the 
Network,  Defeat  the  Device  and  Train  the  Force),  an  evaluation  tool  must  encompass  the 
organization’s  entire  mission.  Therefore,  the  scope  of  this  research  is  not  limited  to  a 
particular  area  of  C-IED  operations.  However,  the  research  is  limited  to  the  mission  of 
JIEDDO  and  the  objectives  of  its  decision  makers.  Lastly,  though  the  focus  of  this  study 
is  IED  defeat  abroad,  the  techniques  and  model  created  could  also  be  applied  to  domestic 
C-IED  defense  issues. 

This  research  will  result  in  a  decision  model  to  be  used  for  selection  of  C-IED 
technologies,  and  application  of  that  model.  It  is  hoped  that  the  study  will  also  offer 
valuable  insight  to  JIEDDO  regarding  their  process  of  soliciting  proposals  from 
contractors  and  assigning  evaluation  experts  to  incoming  proposals. 

The  remainder  of  this  document  is  broken  down  into  four  chapters.  Chapter  2 
provides  a  review  of  relevant  research  that  has  already  been  conducted  in  the  area  of 
proposal  selection  along  with  pertinent  infonnation  regarding  the  literature  on  the  Value- 
Focused  Thinking  methodology  that  is  used.  Chapter  3  provides  detailed  information  on 


the  methodology,  step-by-step,  as  the  research  is  conducted.  Chapter  4  displays  the 
results  of  the  analysis  done  with  the  process  outlined  in  Chapter  3,  and  Chapter  5 
provides  discussion  on  recommendations  based  on  the  results  as  well  as  future  research 
that  can  be  done  to  further  explore  the  issue. 
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II.  Literature  Review 


Decision  Analysis  Overview 

Decision  Analysis  is  a  branch  of  the  Operations  Research  (OR)  discipline.  OR 
lends  itself  quite  readily  to  addressing  decision  problems,  since  the  reason  for  OR  in 
general  is  to  provide  decision  makers  with  a  scientific  method  on  which  to  base  their 
decisions  (Keeney  &  Raiffa,  1993).  Decision  Analysis  includes  the  use  of  procedures, 
methods  and  tools  to  represent  and  assess  a  decision  situation,  and  to  recommend  a 
course  of  action  pursuant  to  the  priorities  of  the  decision  maker  or  stakeholders.  Unaided 
decision  making  works  for  simple  decisions,  but  may  not  suffice  for  complicated  decision 
problems.  Decision  making  relies  on  the  decision  maker’s  intuition,  so  unaided  decision 
making  may  result  in  biases  and  systematic  errors.  Several  studies  have  shown  how  even 
the  simplest  Decision  Analysis  methods  are  superior  to  unaided  intuition.  Currently,  DA 
techniques  are  successfully  used  in  many  private,  public  and  military  applications. 
Alternative  Selection 

The  selection  of  alternatives  is  a  decision  situation  facing  every  decision  maker 
that  is  responsible  for  the  allocation  of  limited  resources.  It  is  a  common  need  among 
these  groups  to  select  the  optimal  solution  or  package  of  solutions.  Often,  there  are 
multiple  interested  parties  and  stakeholders  for  these  decisions,  so  it  is  important  to 
provide  logical  decisions  in  accordance  with  the  organization’s  stated  needs  or  doctrine. 
Approaches  tend  to  be  either  quantitative  or  qualitative,  ranging  from  rigorous  OR 
methods  to  social  science-based  interactive  techniques.  The  family  of  decision  aiding 
techniques  presents  a  wide  range  of  complexity.  Some  approaches  presented  in  the 
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literature  are  so  mathematically  elaborate  that  they  necessitate  the  assistance  of  an  expert 
decision  analyst  in  order  to  be  usable  by  most  real-world  managers  (Henriksen  & 
Traynor,  1999).  Due  to  the  need  of  this  research  effort  to  provide  a  decision  framework 
that  can  easily  be  used  by  JIEDDO  personnel,  the  focus  here  will  be  to  select  a 
methodology  that  provides  a  “user  friendly”  product. 

Decision  Analysis  Methodologies 

There  are  various  techniques  and  methods  available  for  analyzing  decision 
problems.  Most  methodologies  include  the  breaking  down  or  decomposing  the  problem 
to  better  understand  the  situation.  The  general  outline  of  these  methodologies  is  to  define 
the  problem,  identify  objectives  and  examine  alternatives.  Hierarchical  representations 
are  used  for  many  processes  to  illustrate  the  decision  situation.  Two  of  the  most  widely- 
used  decision  making  processes  are  Value -Focused  Thinking  (VFT)  and  the  Analytic 
Hierarchy  Process  (AHP). 

Analytic  Hierarchy  Process 

Although  it  is  not  a  fonnal  subclass  of  the  Decision  Analysis  discipline  (Howard, 
1992),  the  Analytic  Hierarchy  Process  (AHP)  is  an  accepted  technique  for  dealing  with 
complex  decisions.  AHP  consists  of  decomposing  a  decision  problem  into  a  hierarchy 
with  sub-problems  that  can  be  analyzed  independently.  Flexibility  is  emphasized  as  a 
key  capability  of  this  process.  AHP  allows  all  things  relevant  to  the  decision  to  be 
considered  whether  the  aspect  is  tangible  or  intangible,  roughly  estimated  or  precisely 
measured,  etc.  It  is  readily  able  to  deal  with  changes  to  the  priorities  concerning  a 
problem  and  interdependence  among  the  elements  of  a  hierarchy,  while  tracking  the 
logical  consistency  of  judgments  used  in  detennining  priorities  (Saaty  T.  L.,  2001). 
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Saaty  (2001)  also  asserts  that 

An  underlying  assumption  of  AHP  is  that  the  complete  set  of  possible  alternatives 
is  already  available  for  consideration.  The  first  step  of  the  process  is  the  selection 
from  this  possible  group  of  alternatives  for  a  decision  situation.  The  selected 
alternatives  fonn  the  bottom  level  of  the  hierarchy.  Once  this  bottom  level  is 
established,  the  rest  of  the  hierarchy  can  be  constructed.  The  next  level  includes 
the  criteria  by  which  the  alternatives  can  be  measured.  The  priorities  of  these 
criteria  will  be  judged  in  terms  of  their  contribution  to  the  overall  focus  of  the 
hierarchy,  or  the  ultimate  decision  that  must  be  made. 

When  the  criteria  themselves  must  be  measured  in  detail,  a  tier  of  sub  criteria  can 
be  inserted  between  the  parent  criteria  and  the  alternatives.  The  hierarchy  can  be  broken 
down  into  as  many  levels  necessary  to  adequately  represent  the  situation.  Once  this 
initial  model  is  constructed,  it  can  later  be  changed  to  accommodate  new  criteria  or  to 
reflect  changes  in  priorities.  Figure  2  is  a  representation  of  the  basic  AHP  framework. 


Figure  2.  Sample  AHP  Hierarchy  (Colombi,  2007) 


The  next  step  of  AHP  is  to  establish  priorities  for  the  elements  of  the  hierarchy 
through  pairwise  comparisons  of  the  alternatives  against  the  criteria.  Construction  of  a 
matrix  (see  Table  1)  is  the  preferred  process  for  this  analysis.  A  criterion  (C)  is  selected 
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from  the  hierarchy,  then  the  alternatives  (Ai,  A2,. . An)  are  compared  against  this 
property.  Numbers  are  used  to  represent  the  relative  importance  of  one  alternative  over 
another  with  respect  to  the  property.  The  process  for  determination  of  these  numbers  can 
be  seen  as  subjective,  so  it  is  important  to  implement  a  systematic  practice  for  this.  AHP 
uses  a  standard  scale  of  levels  1-9  shown  in  Table  2  to  assign  numeric  judgments  to  each 
element. 

Table  1.  Sample  Matrix  for  Pairwise  Comparison 


c 

Aj 

a2 

A3 

Aj 

1 

1/2 

1/4 

a2 

2 

1 

1/2 

a3 

4 

2 

1 

Total 

7 

3.5 

1.75 
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Table  2.  AHP  Pairwise  Comparison  Scale  (Saaty  T.  L.,  2001) 


Intensity  of 
Importance 

Definition 

Explanation 

1 

Equal  importance  of  both  elements 

Two  elements  contribute 
equally  to  the  property 

3 

Weak  importance  of  one  element  over 
another 

Experience  and  judgment 
slightly  favor  one  element 
over  another 

5 

Essential  or  strong  importance  of  one 
element  over  another 

Experience  and  judgment 
strongly  favor  one  element 
over  another 

7 

Demonstrated  importance  of  one 
element  over  another 

An  element  is  strongly 
favored  and  its  dominance  is 
demonstrated  in  practice 

9 

Absolute  importance  of  one  element 
over  another 

The  evidence  favoring  one 
element  over  another  is  of  the 
highest  possible  order  of 
affirmation 

2,  4,  6,  8 

Intennediate  values  between  two 
adj  acent  judgments 

Compromise  is  needed 
between  two  judgments 

Reciprocals 

If  activity  i  has  one  of  the  preceding 
numbers  assigned  to  it  when  compared 
with  activity  j,  then  j  has  the  reciprocal 
value  when  compared  with  i. 

After  the  assignment  of  numbers,  the  matrix  provides  an  easily  understood 
representation  of  the  relationship  between  each  alternative  and  how  they  fare  against  each 
other  relative  to  the  criterion.  Table  1  is  a  sample  AHP  Matrix  for  pairwise  comparisons 
with  nominal  data.  Note  that  the  diagonal  is  comprised  of  “Is”  due  to  the  need  for  unity 
in  representing  the  comparison  of  an  alternative  to  itself. 

To  obtain  the  set  of  overall  priorities  for  a  decision,  the  results  of  the  pairwise 
comparisons  must  be  synthesized  in  order  to  get  an  overall  estimate  of  the  relative  rank  of 
priorities.  To  do  this,  the  matrix  is  normalized  by  adding  the  values  of  each  column  and 
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then  dividing  each  entry  by  its  column’s  total.  Table  3  illustrates  this  process  with 
nominal  values.  For  simplicity,  the  resulting  fractions  were  put  into  common  terms. 


Tab! 

le  3.  Normalized  Matrix 

C 

Aj 

a2 

a3 

Aj 

1/7 

1/7 

1/7 

a2 

2/7 

2/7 

2/7 

a3 

4/7 

4/7 

4/7 

After  nonnalizing  the  matrix,  the  new  values  in  each  row  are  averaged.  This 
process  reveals  the  percentages  of  overall  relative  preferences: 

A!  =  (1/7  +  1/7  +  1/7)  /  3  =  1/7  =  0. 14 
A2  =  (2/7  +  2/7  +  2/7)  /  3  =  2/7  =  0.29 
A3  =  (4/7  +  4/7  +  4/7)  /3  =  4/7  =  0.57 

So,  in  consideration  of  criterion  C,  Alternative  3  is  about  four  times  more  preferable  than 
Alternative  1  in  this  nominal  example. 

Another  important  consideration  of  AHP  is  consistency.  Judgments  should  have  a 
relatively  high  level  of  consistency,  because  low  consistency  can  give  the  appearance  of 
randomness  among  the  scores  of  the  elements.  The  matrix  in  Table  1  exhibits  good 
consistency  as  is  evident  by  the  reciprocity  of  judgment  scores  for  opposite  comparisons, 
e.g.  A1  vs.  A2  and  A2  vs.  Al.  The  consistency  index  (Cl)  of  a  matrix  is  measured  using 
its  eigenvalue.  A  consistency  ratio  is  then  found  by  comparing  the  matrix’s  Cl  to  that  of 
a  random  matrix  of  the  same  size.  The  accepted  ratio  of  the  random  matrix  Cl  to  the 
hierarchy  matrix  Cl  depends  on  the  size  of  the  matrix  under  consideration  (Saaty  T.  L., 
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2001).  When  the  ratio  exceeds  the  maximum  acceptable  value,  the  scores  within  the 
matrix  or  even  the  overall  structure  of  the  hierarchy  must  be  revisited.  Higher  levels  of 
inconsistency  can  indicate  a  lack  of  accurate  information  and/or  understanding  of  the 
decision  problem. 

Value-Focused  Thinking 

Value-Focused  Thinking  is  another  decision  analysis  method  designed  to  assist  in 
complex  decision  situations  with  multiple  objectives  and  priorities.  The  first  fundamental 
difference  between  VFT  and  the  previously  discussed  AHP  is  the  generation  of 
alternatives.  Whereas  the  identification  of  alternatives  is  the  basis  of  the  AHP  hierarchy, 
alternatives  are  not  considered  until  the  latter  steps  of  the  VFT  process. 

Value-Focused  Thinking  consists  of  two  overriding  activities:  first  deciding  what 
you  want,  and  then  figuring  out  how  to  get  it.  A  significant  effort  is  allocated  to 
articulating  values  in  the  early  steps  of  VFT.  This  method  incorporates  the  decision 
context  into  values  based  on  what  the  Decision  Maker  (DM)  wants  to  achieve.  VFT 
analysis  is  ideally  done  with  a  single  DM  who  is  easily  accessible  and  eager  to  participate 
in  the  process.  Unfortunately,  this  is  often  not  possible,  and  other  sources  must  be 
consulted  for  elicitation  of  values,  which  is  often  the  most  difficult  part  of  the  VFT 
process. 

There  are  three  standards  of  sources  for  the  determination  of  values:  platinum, 
gold,  and  silver.  Platinum  is  the  most  preferred  because  interviews  with  senior 
stakeholders  and  the  actual  decision  makers  are  used  to  detennine  objectives.  Gold  is 
slightly  less  preferred  and  uses  published  policy  or  strategic  planning  documents 
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approved  by  the  DMs.  The  least  desirable  is  silver  which  is  limited  to  interviews  with 
subject  matter  experts  (SME)  and  representatives  of  the  stakeholders  (Knighton,  2007). 

Once  the  values  have  been  determined,  the  hierarchy  can  be  created.  Whereas  the 
AHP  process  uses  a  bottom-up  approach  for  value  structuring,  VFT  employs  a  top-down 
approach  which  starts  with  an  overall  value  and  then  subdivides  this  value  into  successive 
levels  until  measurable  values  have  been  reached.  Kirkwood  (1997)  outlines  a  few 
desirable  properties  of  value  hierarchies  that  are  especially  important  when  using  a  VFT 
approach.  Hierarchies  must  be  collectively  exhaustive  and  mutually  exclusive.  That  is, 
the  evaluation  considerations  in  each  layer  must  include  everything  needed  to  evaluate 
the  decision  alternatives,  and  nothing  necessary  to  do  the  evaluation  can  be  included  in 
more  than  one  evaluation  consideration  (Kirkwood,  1997).  Unlike  an  AHP 
representation,  a  VFT  model  must  exhibit  independence  among  elements  of  the 
hierarchy.  This  ensures  that  changing  the  value  of  one  consideration  does  not  cause  a 
change  in  a  value  of  another.  Since  VFT  employs  an  additive  value  function  for  the 
scoring  of  alternatives,  independence  among  values  becomes  very  important.  Finally, 
complexity  and  large  size  should  be  avoided  so  that  the  intended  audience  can  easily 
understand  and  use  the  hierarchy  (Kirkwood,  1997).  Figure  3  is  a  representation  of  the 
basic  VFT  hierarchical  framework. 
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Figure  3.  Sample  VFT  Hierarchy  (Kirkwood,  1997) 


The  next  step  of  the  VFT  process  is  to  create  evaluation  measures.  Each  of  the 
lowest-level  values  in  the  hierarchy  will  be  measured  for  each  alternative.  Evaluation 
measure  scales  can  be  classified  as  either  natural  or  constructed,  and  also  as  either  direct 
or  proxy  (Kirkwood,  1997).  A  natural  scale  is  one  that  is  general  use  and  has  a  common 
interpretation,  for  example,  time,  whereas  a  constructed  scale  is  developed  for  a 
particular  decision  problem.  A  direct  scale  directly  measures  the  degree  of  attainment  of 
an  objective  and  a  proxy  scale  reflects  attainment  of  the  goal,  but  does  not  directly 
measure  it.  The  most  desirable  situation  is  to  have  a  natural-direct  scale,  but  this  is  often 
not  possible  in  real-world  decision  situations.  Table  4  shows  the  preferences  among  the 
various  combinations  of  scales. 
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Table  4.  Scale  Order  of  Preference 


Natural 

Constructed 

Direct 

Best 

Acceptable 

Proxy 

Acceptable 

Least 

Desirable 

Now,  the  process  calls  for  a  mathematical  translation  of  each  measure.  This  is 
accomplished  by  using  a  single-dimensional  value  function  (SDVF)  as  defined  in  VFT 
literature.  In  the  Systems  Engineering  community,  these  are  referred  to  as  utility  curves 
(Buede,  2000).  The  benefit  of  developing  SDVFs  is  that  all  evaluation  measures  are 
transfonned  into  comparable  units  that  can  be  weighed  against  each  other.  SDVFs  are 
either  continuous  (linear,  piecewise  linear,  or  exponential)  or  discrete  (categorical  or  by 
value  increments),  and  must  be  either  monotonically  increasing  or  monotonically 
decreasing  (Kirkwood,  1997).  This  can  be  seen  in  the  example  SDVFs  in  Figure  4. 


Categorical 


Exponential 


Piecewise  Linear 


Figure  4.  Single  Dimensional  Value  Function  Examples 
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After  SDVFs  have  been  established,  the  elements  of  the  hierarchy  must  be 
weighted.  Weighting  allows  the  DM  to  give  relative  importance  to  changes  in  evaluation 
measures.  For  this  study,  the  technique  of  swing  weighting  was  used.  Swing  weighting 
is  accomplished  by  detennining  the  importance  of  “swinging”  an  evaluation  measure 
from  the  least  preferred  value  to  the  most  preferred,  placing  the  “swings”  in  increasing 
order  of  importance,  and  finally  expressing  the  multiple  of  all  the  value  swings  in  terms 
of  multiples  of  the  least  important  swing.  All  global  weights  on  the  lowest  tier  must  sum 
to  one,  as  well  as  all  local  weights  on  the  same  tier. 

The  next  step  in  the  VFT  process  is  the  scoring  of  a  set  of  alternatives  according 
to  the  following  equation: 

Value(X)  =  T,f=1wivi(xi')  (1) 

where 

Value(X)  =  the  overall  score  of  an  alternative 
v,{x,)  =  the  value  of  the  score  on  the  i,b  measure 
Wi  =  the  weight  of  the  i,h  measure 
n  =  the  total  number  of  measures 

Analysis  is  then  conducted  by  ranking  the  scores  of  all  alternatives  in  the  set  under 
consideration.  Alternatives  with  higher  scores  are  preferred. 

After  all  the  alternatives  in  the  set  have  been  scored,  sensitivity  analysis  is 
conducted  to  detennine  the  impact  on  the  ranking  of  alternatives  when  changes  are  made 
to  various  model  assumptions.  A  common  sensitivity  analysis  of  interest  is  that  of  the 
weights  (Kirkwood,  1997).  Since  weighting  of  particular  values  can  be  a  matter  of 
disagreement  among  stakeholders,  it  becomes  necessary  to  evaluate  the  weights  on  these 
value  to  determine  whether  or  not  the  scores  of  the  alternatives  are  sensitive  based  on  the 
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weight  of  a  particular  value  measure.  Sensitivity  to  weighting  is  measured  by  ranging  the 
value  of  a  certain  measure  from  zero  to  one  and  examining  the  results  in  comparison  to 
the  original  weight.  If  the  overall  rankings  of  the  alternatives  remain  unchanged  by 
varying  this  weight,  then  it  can  be  determined  that  the  measure  is  insensitive.  A  change 
to  the  ranking  would  indicate  sensitivity,  and  a  need  to  further  examine  the  weighting  of 
the  measure  in  question  with  the  decision  makers.  Finally,  conclusions  and 
recommendations  are  made  to  the  stakeholders  regarding  the  findings  of  the  model. 
Methodology  Selection  in  Published  Studies 

There  are  many  examples  in  the  literature  of  the  application  of  DA  to  the  analysis 
and  selection  of  alternatives.  A  few  will  be  discussed  here  with  the  goal  of  comparing 
and  contrasting  the  particular  decision  situations  and  examining  how  they  may  relate  to 
JIEDDO’s  decision  problem. 

Trumm’s  2006  study  details  the  implementation  of  a  decision  tool  to  aid  in  the  Air 
Force’s  Source  Selection  Process.  Source  selection  is  an  extremely  detailed  and  time 
consuming  course  of  action  used  to  select  the  best  offeror  responding  to  a  specific 
solicitation.  Prior  to  the  study,  the  source  selection  process  for  choosing  a  contractor  did 
not  incorporate  a  standardized  objective  decision  analysis  tool;  therefore,  the  process  was 
extremely  subjective  and  provided  little  guidance  to  distinguish  between  highly 
competitive  contractors  (Trumm,  2006). 

There  was  published  Air  Force  guidance  on  the  selection  of  contractors,  but  this 
was  largely  based  on  price  for  the  proposed  job  and  the  contractor’s  past  performance.  A 
Value-Focused  Thinking  decision  model  was  developed  using  the  existing  documentation 
and  collective  source  selection  expertise  of  a  base  civil  engineering  organization.  This 
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organization  cooperated  with  researchers  to  detennine  measures  for  contractor 
evaluation.  In  addition  to  the  pre-established  price  and  past  performance  measures, 
technical  excellence,  management  capability,  financial  capability  and  personnel 
qualifications  were  detennined  to  be  important  enough  to  include  in  the  analysis.  The 
model  was  developed,  contractors  were  scored,  and  the  model  was  refined  through 
several  iterations. 

Trumin  determined  at  the  end  of  his  study  that  Decision  Analysis,  using  VFT  in 
particular,  is  a  viable  and  essential  tool  for  the  source  selection  process.  There  were 
several  problems  with  the  existing  source  selection  process,  but  VFT  was  shown  to 
remedy  these  problems  through  the  objective  analysis  process  described  in  the  research. 

Another  example  of  a  decision  aid  methodology  as  it  relates  to  a  selection  process 
is  Boeg  and  Koett’s  1998  study  conducted  for  the  Air  Force  Research  Laboratory 
(AFRL).  In  this  case,  the  Analytic  Hierarchy  Process  was  used  to  aid  in  requirements 
analysis  and  to  then  evaluate  competing  system  proposals.  The  development  of 
requirements  for  a  large  system  is  well  suited  to  a  top-down  hierarchy  model.  High  level 
mission  or  system  needs  are  identified  and  then  decomposed  into  lower  level  operational 
requirements  for  the  proposed  system.  This  closely  follows  the  hierarchy  construction 
proposed  in  AHP.  Cost  was  a  driving  factor  in  the  study,  which  produced  issues  with 
interdependence  within  the  elements  of  the  hierarchy.  AHP  is  equipped  to  deal  with  the 
interdependency  whereas  VFT  is  not  (Kott,  Boag,  &  Vargas,  1998).  Another  reason  for 
the  choice  to  use  AHP  for  the  study  was  its  application  regarding  the  comparison  of 
alternatives.  Here,  a  finite  set  of  two  alternatives  was  submitted  which  could  be  easily 
compared  to  one  another  as  the  lowest  level  of  the  hierarchy.  The  two  system  proposals 
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were  compared  in  tenns  of  their  ability  to  satisfy  each  of  the  leaf  level  requirements.  The 
sum  of  the  contributions  determined  the  overall  ranking  of  the  solution. 

The  focus  of  the  study  was  a  technique  that  could  be  applied  to  the  different 
phases  of  the  requirements  definition  process,  and  ultimately  aid  in  the  evaluation  of 
alternatives.  AHP  was  selected  for  this  analysis  because  it  provided  a  consistent  and  low- 
effort  method  for  estimating  the  relative  impact  of  each  requirement  on  the  system’s 
overall  cost,  benefits  and  risk  (Kott,  Boag,  &  Vargas,  1998).  Again,  given  the  finite  set 
of  alternatives,  AHP  was  well  suited  to  determining  the  best  system  proposal  for 
selection. 

Henriksen  and  Traynor’s  1999  study  focused  on  the  establishment  of  a  research 
and  development  project  selection  process  based  on  the  relative  value  to  the  organization 
of  the  proposed  research.  This  study  seems  to  reflect  a  combination  of  the  aspects  of  a 
few  different  decision  methodologies.  After  reviewing  many  existing  processes,  it  was 
determined  that  a  practical  model  was  needed  that  would  be  easy  for  decision  makers  to 
understand  and  use.  This  was  important  for  the  sake  of  simplicity  as  well  as  to  bolster 
support  for  the  proposed  process. 

The  improved  scoring  technique  realized  in  the  Henriksen  study  incorporated 
tradeoffs  among  evaluation  criteria.  The  resulting  figure  of  merit  was  then  combined 
with  a  scaled  funds  request  to  obtain  a  value  index  for  each  proposed  project.  The  value 
index  algorithm  produced  a  measure  of  project  value  that  accounted  for  value  as  a 
function  of  both  merit  and  cost.  It  was  shown  that  a  combined  additive/multiplicative 
algorithm  correctly  discriminated  the  desired  characteristics  and  that  proposals  were 
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ranked  consistently  with  the  institution’s  intended  emphasis.  (Henriksen  &  Traynor, 
1999) 

Like  a  Value-Focused  Thinking  approach,  this  model  was  initially  based  on  the 
values  of  survey  respondents.  However,  it  departs  from  VFT  in  the  use  of  an 
additive/multiplicative  function  for  scoring  of  proposals.  Though  this  method  employs  a 
multiplicative  aspect,  it  is  unlike  the  AHP  process  in  than  it  does  not  prescribe  the 
existence  of  a  pre-existing  finite  set  of  alternatives.  In  this  respect,  the  process  more 
resembles  VFT.  It  should  be  noted,  though,  that  the  goal  of  the  research  was  not  to 
simply  rank  a  list  of  alternatives,  but  to  promote  cost  effectiveness  and  maximum  utility 
in  R&D  activities.  The  result  of  the  study  revealed  the  30  top  scoring  projects  of  a 
candidate  group.  Of  note,  only  16  of  the  30  had  actually  been  selected  for  funding.  The 
research  determined  that  decision  makers  are  ultimately  responsible  for  the  creation  of 
the  best  portfolio  of  alternatives  to  select  for  funding,  and  that  there  certainly  may  be 
added  value  from  various  combinations  of  alternatives  that  would  not  be  revealed  if  only 
the  top  scoring  proposals  were  chosen  (Henriksen  &  Traynor,  1999). 

The  current  study  commissioned  by  JIEDDO  was  prompted  by  a  classified  study 
conducted  at  the  Air  Force  Institute  of  Technology  in  2007.  This  study  used  the  Joint 
Capabilities  Integration  and  Development  System  (JCIDS)  to  identify  and  prioritize 
alternative  solutions  to  C-IED  capabilities.  JCIDS  is  the  early  Department  of  Defense 
(DoD)  life  cycle  analysis  process  that  refines  warfighting  concepts,  requirements  and 
evaluation  criteria  for  future  defense  programs  (Chainnan  of  the  Joint  Chiefs  of  Staff, 
2007).  The  objective  of  the  2007  AFIT  study  was  to  recommend  an  optimal  combination 
of  technology  initiatives  to  Air  Combat  Command  (ACC)  for  funding. 
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As  with  many  other  DA  methodologies,  an  objectives  hierarchy  was  created  for 
the  2007  AFIT  study.  That  research  effort  was  scoped  to  focus  on  materiel  alternatives 
only,  due  to  the  difficulty  in  assessing  materiel  and  non-materiel  solutions  using  the  same 
evaluation  model.  (Of  note,  this  assessment  challenge  was  also  encountered  by  the 
current  research  effort.)  The  process  for  constructing  the  hierarchy  was  top-down  and 
alternative  based.  Values  and  weights  were  used  to  determine  a  priority  and  competing 
technologies  were  ranked  against  the  priority. 

The  current  research  builds  on  the  alternative  prioritization  developed  by  the 
previously  mentioned  AFIT  study.  However,  rather  than  follow  the  JCIDS  process  to 
develop  a  one-time  prioritized  list,  this  paper  will  describe  an  IED  defeat  Decision 
Analysis  model  that  can  be  used  repeatedly  to  evaluate  successive  alternative  sets. 
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III.  Methodology 


Methodology  Selection  for  the  JIEDDO  Decision  Problem 

Value-Focused  Thinking  was  detennined  to  be  the  best  fit  for  the  C-IED  decision 
problem.  However,  there  is  a  vigorous  debate  in  the  literature  between  proponents  of 
AHP  and  Decision  Analysis  (of  which  VFT  is  a  part)  as  to  the  effectiveness  of  each 
method.  Therefore,  the  key  points  of  the  debate  are  summarized  here  in  order  to  provide 
a  framework  for  the  selection  of  VFT  as  the  research  methodology. 

Decision  Analysis  practitioners  question  the  theoretical  underpinnings  of  AHP 
and  assert  that  the  application  of  AHP  in  certain  cases  allows  decision  makers  to  make 
politically  or  economically  expedient  decisions  while  still  claiming  the  “pedigree”  of  a 
formal  decision  process  (Howard,  1992).  Critics  of  DA,  by  contrast,  claim  that  it  is  an 
outmoded  technique  of  limited  usefulness,  whose  axioms  do  not  adequately  take  into 
account  the  situational  constraints  faced  by  decision  makers  (Brown,  1992). 

One  of  the  biggest  areas  of  debate  is  that  of  the  principle  of  hierarchical 
composition — the  requirement  that  evaluation  criteria  be  independent  of  the  alternatives 
considered.  The  DA  community  believes  that  AHP  violates  this  principle,  thus  leading  to 
the  phenomenon  of  rank  reversal  when  new  alternatives  are  added  to  a  set  of  previously 
scored  alternatives  (Dyer,  1990). 

AHP  experts  counter  this  claim  by  stating  that  alternatives  do  in  fact  impact  the 
criteria,  and  therefore  rank  reversal  should  be  expected.  Based  on  Saaty’s  response 
(1990)  to  Dyer’s  criticism,  a  JIEDDO-related  example  of  rank  reversal  might  be  the 
portfolio  management  aspect  of  fielding  C-IED  technology — as  more  systems  to  counter 
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a  certain  type  of  IED  threat  are  fielded,  the  capability  gap  narrows  and  the  value  of  each 
individual  system  intended  to  address  that  gap  decreases.  Saaty’s  position  is  that  the 
addition  of  alternatives  changes  the  selection  criteria  and  consequently  the  ranking. 
Therefore,  rank  reversal  is  a  logical  outcome  of  the  process  and  not  a  fallacy  in  the 
underlying  theory.  AHP  addresses  the  rank  reversal  problem  through  the  use  of  an 
extensive  supermatrix  that  requires  a  large  number  of  pairwise  comparisons  on  the  part  of 
the  decision  maker  (Dyer,  1990). 

Of  note,  it  is  not  feasible  for  a  JIEDDO  decision  maker  to  perform  large  numbers 
of  pairwise  comparisons  every  time  a  new  alternative  is  considered.  However,  the 
proposed  decision  model  handles  the  supply  and  demand  issue  without  the  aid  of  AHP  by 
incorporating  JIEDDO ’s  existing  gap  prioritization  process  into  the  value  hierarchy.  As 
the  gaps  are  reassessed  on  a  periodic  basis  due  in  part  to  portfolio  management 
considerations,  candidate  technologies  are  evaluated  on  their  ability  to  meet  the  currently 
prioritized  gaps. 

There  are  advantages  and  disadvantages  to  the  application  of  every  decision 
analysis  tool  to  any  real-world  decision  problem,  and  the  two  presented  here  are  no 
exception.  AHP  is  a  very  mathematically  structured  approach  which  is  desirable  for  the 
defensibility  of  the  process,  but  presents  a  major  difficulty  in  its  application  to  JIEDDO’ s 
decision  problem.  Since  C-IED  proposals  are  submitted  to  JIEDDO  on  a  continuous 
basis,  there  will  never  be  a  finite  set  of  alternatives  to  compare,  which  is  key  to  the  AHP 
methodology.  In  addition,  while  AHP  claims  a  theoretical  basis  (the  supermatrix)  for 
addressing  the  supply  and  demand  issue  of  portfolio  management,  the  implementation  of 
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the  VFT  model  proposed  by  this  paper  in  conjunction  with  periodic  gap  prioritization  will 
accomplish  the  same  goal. 

Ultimately,  a  VFT  hierarchy  is  preferable  since  it  allows  for  an  unlimited  number 
of  alternatives  to  be  evaluated  in  batch  sizes  determined  by  JIEDDO,  without  the 
necessity  of  making  instantaneous  modifications  to  the  hierarchy  as  new  alternatives 
enter  the  evaluation  cycle.  A  possible  use  of  AHP  would  be  at  the  sustain  stage  of  the 
JCAAMP  process  where  alternatives  are  known  and  need  to  be  evaluated  against  one 
another  for  determining  continuation  of  funding. 

Development  of  the  Decision  Model 

After  VFT  was  selected  as  the  preferred  methodology,  the  first  step  in  building 
the  value  hierarchy  was  the  elicitation  of  DM  values,  which  was  performed  using  three 
different  methods.  First,  the  research  team  attended  JIEDDO's  October  2007  Technology 
Outreach  Conference  to  obtain  an  accurate  perspective  of  the  dynamics  of  the  counter- 
IED  mission  area.  Then,  the  team  also  solicited  values  directly  from  JIEDDO  personnel 
and  finally  researched  the  organization's  source  documents. 

After  obtaining  the  necessary  background  information,  an  affinity  diagram 
grouping  exercise  was  performed  and  a  comprehensive  value  hierarchy  was  constructed 
on  a  whiteboard.  The  hierarchy  was  then  developed  electronically  using  Hierarchy 
Builder  software  developed  by  Lt  Col  Jeffery  Weir  (Weir,  2008)  and  is  depicted  in 
Figure  5. 
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Figure  5.  IED  Defeat  Value  Hierarchy 


Delineation  of  Individual  Values 

Potential  to  Defeat  IEDs  was  chosen  as  the  overall  value  for  the  hierarchy  for 
obvious  reasons.  The  second  tier  values  are  Needed  Capability,  Operational  Performance 
and  Usability.  Needed  Capability  addresses  how  useful  the  solution  will  be  in  the  current 
threat  environment.  Operational  Performance  assesses  how  well  the  solution  meets  the 
criteria  for  its  intended  task.  Usability,  as  the  name  implies,  reflects  ease  of  use  for  the 
end  user.  Following  are  descriptions  of  the  leaf-level  values  within  each  of  the  three 
branches: 
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Tenets  Impacted  (Needed  Capability)  answers  the  question,  “Does  this  proposal 
impact  one  or  many  tenets?”  This  value  allows  the  hierarchy  to  capture  the  synergistic 
value  of  a  solution  that  impacts  more  than  one  tenet. 

Gap  Impact  (Needed  Capability):  JIEDDO  establishes  prioritized  capability  gaps 
on  a  periodic  basis  with  input  from  the  Combatant  Commands.  Thus,  the  value  of  a 
proposal  is  directly  related  to  the  priority  of  the  gap  which  it  targets. 

Classification  (Needed  Capability):  How  easily  can  this  solution  be  shared 
among  stakeholders  within  and  outside  of  DoD? 

Time  to  Counter  (Needed  Capability):  How  long  will  it  take  for  the  enemy  to 
develop  a  counter-measure  for  the  system? 

Technical  Performance  (Operational  Performance):  The  predicted  performance 
of  a  system  while  executing  its  intended  mission. 

Suitability  (Operational  Performance):  How  well  the  system  will  perform  in  its 
intended  environment. 

Interoperability  (Operational  Performance)  is  the  degree  to  which  a  system  fits 
into  the  existing  network  architecture,  whether  it  can  exchange  data  with  supporting  and 
supported  systems,  and/or  whether  it  can  perform  its  task  without  negatively  impacting 
friendly  assets. 

Technical  Risk  (Operational  Performance):  JIEDDO  is  a  risk- tolerant 
organization.  They  are  willing  to  accept  technology  risk  if  it  is  outweighed  by  other 
benefits.  However,  a  mature  technology  will  provide  more  value  than  an  unproven 
technology  for  the  same  performance. 
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Fielding  Timeline  (Operational  Performance):  How  soon  the  solution  can  be 
fielded.  If  a  solution  can’t  be  fielded  in  a  timely  manner  it  becomes  much  less  relevant  to 
JIEDDO — only  10%  of  their  budget  is  spent  on  proposals  with  a  timeline  of  3  years  or 
longer.  “Fielded”  is  defined  as  when  the  first  article  of  a  system  is  delivered. 

Operations  Burden  (Usability):  The  degree  to  which  the  system  will  impact  the 
capacity  of  its  host  environment,  e.g.  bandwidth  required  by  a  collaborative  software 
system,  weight  for  a  vehicle  or  soldier  mounted  system,  or  rack  space  required  for  server 
enabled  analysis  system. 

Work  Load  (Usability):  This  value  captures  the  time  requirements  that  the 
solution  places  on  the  user  in  an  operational  environment  to  ensure  that  the  system 
continues  to  operate  as  expected. 

Required  Training  (Usability):  This  value  refers  to  the  organic  training 
requirements  of  each  solution  and  should  not  be  confused  with  the  LOO  of  “Train  the 
Force”.  It  contains  two  sub  values: 

Training  Time:  How  long  does  it  take  to  train  the  average  user  on  the 

solution? 

Training  Program  Maturity:  How  well  does  the  solution’s  training 
capability  reflect  the  actual  system  when  used  in  its  intended  environment? 

Weighting  of  Values 

After  the  value  hierarchy  was  constructed,  the  values  were  weighted  in  a  process 
known  as  swing  weighting.  This  is  done  considering  all  values  below  a  single  upper  tier 
value  and  “swinging”  each  value  individually  from  its  least  preferred  to  most  preferred 
levels.  The  importance  of  swinging  each  value  is  assessed  relative  to  the  other  values, 
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and  the  values  are  ranked  from  least  to  most  important.  The  weight  of  each  value  is  then 
expressed  as  a  multiple  of  “k”,  the  weight  of  the  least  important  value. 

Top  Tier  Weighting 

The  C-IED  mission  area  is  a  fluid  operational  environment.  Therefore,  the 
overriding  priority  is  to  develop  proposals  that  target  the  current  needs  of  the  C-IED 
warfighter.  A  system  that  does  an  adequate  job  and  addresses  an  urgent  need  is 
preferable  to  a  system  that  perfectly  fills  a  low  priority  or  nonexistent  capability  gap. 
Therefore  Needed  Capability  was  ranked  as  the  most  important  of  the  top  tier  values. 
Operational  Perfonnance  was  weighted  as  the  next  most  important  because  the  system 
needs  to  be  able  to  do  the  job  it  was  designed  to  do  in  its  intended  environment. 


r 

fable  5.  Top  Tier  Weighting 

Value 

k  value 

Weight 

Needed  Capability 

1.6k 

.4 

Operational  Performance 

1.4k 

.35 

Usability 

k 

.25 

Needed  Capability  Weighting 

The  fluid  nature  of  the  C-IED  battlespace  places  a  premium  on  JIEDDO’s  ability 
to  fund  initiative  portfolios  that  are  precisely  targeted  at  the  warfighter’s  most  current 
capability  gaps.  Therefore,  the  most  weight  was  placed  on  Gap  Impact.  The  next  most 
highly  weighted  value  was  Time  to  Counter,  because  there  is  relatively  little  benefit  to 
funding  a  proposal  that  will  be  quickly  rendered  irrelevant  on  the  battlefield  by  enemy 
countermeasures.  Lastly,  the  least  weight  was  placed  on  Tenets  Impacted  and 


32 


Classification.  A  proposal  can  still  be  of  some  value  even  if  it  does  not  impact  multiple 
tenets  or  is  highly  classified. 


Table  6.  Needed  Capability  Weigl 

iting 

Value 

k  value 

Weight 

Gap  Impact 

3k 

.44 

Time  to  Counter 

2k 

.28 

Tenets  Impacted 

k 

.14 

Classification 

k 

.14 

Operational  Performance  Weighting 

Technical  Performance  was  determined  to  be  the  most  important  value  within 
Operational  Perfonnance.  The  most  critical  aspect  of  a  device  or  system  within  this 
context  is  that  it  is  effective  in  the  completion  of  its  intended  mission.  The  next  most 
highly  weighted  value  was  Interoperability,  because  one  of  the  most  pressing  problems  in 
the  C-IED  environment  is  the  large  number  of  systems  that  interfere  or  are  unable  to 
communicate  with  each  other  (Jacoby,  2008).  Suitability  and  Fielding  Timeline  value 
were  weighted  next  due  to  the  requirement  to  rapidly  field  systems  that  can  operate 
effectively  in  their  intended  environment.  The  team  assigned  the  lowest  weight  to  the 
value  of  Technical  Risk  because  JIEDDO  is  a  risk-tolerant  organization. 
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Table  7.  Operational  Performance  Weighting 


Value 

k  value 

Weight 

Technical  Performance 

3k 

.315 

Interoperability 

2.5k 

.26 

Suitability 

1.5k 

.16 

Fielding  Timeline 

1.5k 

.16 

Technical  Risk 

k 

.105 

Usability  Weighting 

Work  Load  was  determined  to  be  the  most  important  value  due  to  its  potential 
impact  on  situational  awareness  in  a  combat  environment.  Operations  Burden  was 
weighted  the  next  highest  because  it  impacts  users  every  time  they  use  the  system  over  its 
entire  lifetime.  By  contrast,  Required  Training  was  determined  to  have  the  lowest  value 
because  the  timeline  of  its  impact  is  limited  to  the  initial  familiarization  phase. 


Table  8.  Usability  Weighting 


Value 

k  value 

Weight 

Work  Load 

1.6k 

.40 

Operations  Burden 

1.4k 

.35 

Required  Training 

k 

.25 

Required  Training  Weighting 

Training  Time  was  detennined  to  be  more  important  than  Training  Program 
Maturity.  It  is  more  desirable  to  use  a  “good  enough”  training  program  to  quickly 
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familiarize  a  soldier  with  a  system  than  to  use  a  “perfect”  program  that  takes  an  excessive 
amount  of  training  time  and  delays  the  soldier’s  deployment  to  theater. 


Table  9.  Required  Training  Weigl 

hting 

Value 

k  value 

Weight 

Training  Time 

4k 

.8 

Training  Program 

Maturity 

k 

.2 

Single  Dimensional  Value  Functions  (SDVFs)  and  Evaluation  Measures  and  Methods 
SDVFs,  evaluation  measures  and  evaluation  methods  were  developed  for  each 
leaf  level  value  and  are  described  in  detail  below.  A  combination  of  categorical, 
exponential  and  piecewise  linear  SDVFs  was  used.  All  measures  were  direct  measures 
and  used  both  natural  and  constructed  scales. 

Many  people  and  organizations  will  be  involved  in  determining  inputs  to  the 
SDVFs  for  each  proposal.  The  JIEDDO  Initiatives  and  Technology  Branch  (ITB)  can 
determine  inputs  for  those  SDVFs  that  do  not  require  specific  expertise,  such  as  Tenets 
Impacted.  BIDS  evaluators  are  experts  in  their  field  and  can  be  used  for  technology¬ 
centric  SDVFs  such  as  Technical  Risk.  Operations  subject  matter  experts  are  familiar 
with  the  counter-IED  mission  area  and  current  developments  in  theater  and  can  best 
assess  values  such  as  Technical  Performance.  The  JIEDDO  Technical  Gaming  Team 
(TGT)  is  composed  of  experts  in  enemy  tactics,  techniques  procedures  and  capabilities 
and  can  assess  proposals'  Time  to  Counter. 

Tenets  Impacted  is  assessed  using  a  categorical  increasing  value  function  based 
on  a  natural  scale  of  the  number  of  positively  impacted  tenets.  The  JIEDDO  ITB  could 
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determine  how  many  tenets  are  targeted  by  the  proposal.  Since  a  proposal  must  target  at 
least  one  tenet  to  be  considered,  50%  of  the  value  is  awarded  for  addressing  this 
requirement.  Some  proposals  target  more  than  one  tenet,  so  additional  value  is  awarded 
for  addressing  two  or  three  tenets.  It  is  not  likely  that  a  proposal  will  address  more  than 
three  tenets,  so  the  maximum  value  is  awarded  at  this  level. 


Figure  6.  Tenets  Impacted  SDVF 


Gap  Impact  is  represented  by  a  categorical  increasing  SDVF,  with  a  constructed 
scale  of  the  prioritized  gaps.  The  JIEDDO  ITB  could  determine  which  gap  is  targeted  by 
each  proposal,  if  any,  and  value  assigned  to  each  proposal  based  on  the  rank  of  the  gap  it 
targets.  Full  value  is  awarded  for  targeting  the  highest  priority  gap,  with  value 
measurements  linearly  decreasing  through  Gaps  #8  and  below,  which  yield  a  value  score 
of  0.22. 
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Figure  7.  Gap  Impact  SDVF 


Classification  follows  a  categorical  SDVF  using  the  natural  scale  of  Classification 
Level,  which  will  be  determined  by  the  proponent  and  validated  by  the  ITB.  There  is  a 
return  to  scale  of  .2  to  .5  when  moving  from  the  SECRET/NOFORN  to  SEC/REL  levels 
due  to  the  necessity  to  share  as  much  as  information  as  possible  with  our  coalition 
partners,  within  the  bounds  of  national  security.  Not  being  able  to  share  classified 
information  hampers  cooperation  at  all  levels  of  warfare. 


Figure  8.  Classification  SDVF 
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Time  to  Counter  is  represented  by  an  increasing  exponential  SDVF  measured  on  a 
natural  scale  expressed  in  months.  A  proposal  must  have  a  projected  time  to  counter  of 
50  months  to  get  half  the  available  value,  because  it  is  desirable  to  develop  and  field 
systems  that  will  provide  maximum  return  on  the  investment  of  time  and  money,  and 
minimize  the  requirement  for  end  users  to  learn  yet  another  C-IED  system.  The  SDVF's 
input  would  be  based  on  a  preliminary  assessment  conducted  by  the  JIEDDO  Technical 
Gaming  Team  (TGT).  The  SDVF  exponential  is  defined  by: 

X 

Value  (x)  =  (2) 


Figure  9.  Time  to  Counter  SDVF 


Technical  Performance  is  another  categorical,  linearly  increasing  SDVF  based  on 
a  constructed  scale  (see  Table  10).  BIDS  evaluators  could  assign  a  performance  rating  to 
a  proposal  based  on  how  well  the  system  is  predicted  to  meet  the  requirements  of  its 
targeted  gap  if  it  functions  as  advertised  by  the  proponent. 
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1  2  3  4  5 

Technical  Performance  Rating 


Figure  10.  Technical  Performance  SDVF 


Table  10.  Technical  Performance  Assessment  Scale 


1 

Provides  none  of  the  capability  required  by  the  intended  mission. 

2 

Provides  some  of  the  capability  required  by  the  intended  mission. 

3 

Provides  much  of  the  capability  required  by  the  intended  mission. 

4 

Provides  most  of  the  capability  required  by  the  intended  mission. 

5 

Provides  all  of  the  capability  required  by  the  intended  mission. 

Suitability  is  a  categorical,  linearly  increasing  SDVF  based  on  a 
constructed  scale  (see  Table  1 1).  The  suitability  rating  could  be  assessed  by  an 
operations  SME  by  comparing  the  system  performance  parameters  provided  by  the 
proponent  with  the  conditions  likely  to  be  encountered  by  the  system  in  its  area  of 
intended  use. 
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Figure  11.  Suitability  SDVF 

Table  11.  Suitability  Assessment  Scale 


1 

System  performance  would  be  critically  degraded  in  operational  environment. 

2 

System  performance  would  be  significantly  degraded  in  operational  environment. 

3 

System  performance  would  be  moderately  degraded  in  operational  environment. 

4 

System  performance  would  be  slightly  degraded  in  operational  environment. 

5 

System  performance  would  not  be  degraded  in  operational  environment. 

Interoperability  is  another  a  categorical,  linearly  increasing  SDVF  based  on  a 
SME  assessment  of  the  level  of  interoperability  issues  likely  to  be  associated  with 
integrating  the  new  system  into  the  current  C-IED  architecture.  Selection  of  a  particular 
SME  (electronic  warfare,  data  link,  etc.)  will  be  determined  by  the  technology  of  the 
system  being  evaluated.  Full  value  is  awarded  for  those  systems  that  can  be  seamlessly 
integrated,  while  50%  of  the  available  value  is  awarded  for  systems  with  only  minor 
interoperability  issues. 
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Figure  12.  Interoperability  SDVF 


Table  12.  Interoperability  Assessment  Scale 


Significant 

Incompatible  system  links  with  current  architecture,  or 
Electromagnetic  Interference  (EMI)  with  other  systems  that 
is  not  easily  overcome. 

Minor 

Incompatibility  issues  that  are  easily  overcome  by  minor 
changes  in  system  design. 

None 

No  issues  affecting  interoperability  of  proposal. 

Technical  Risk  is  measured  using  a  categorical  SDVF  based  on  the  constructed 
but  widely  used  Technology  Readiness  Level  (TRL)  scale,  located  at  Appendix  A.  Since 
JIEDDO  requires  that  proposals  be  at  a  TRL  6  or  higher  before  proceeding  from  the 
proposal  assessment  to  initiative  evaluation  stage,  there  is  a  significant  return  to  scale— 
from  0.3  to  0.9— when  moving  from  TRL  5  to  TRL  6.  A  designated  BIDS  evaluator 
could  assess  Technology  Readiness  Level. 
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Technology  Readiness  Level 


Figure  13.  Technical  Risk  SDVF 

Fielding  Timeline  is  assessed  by  a  decreasing  piecewise  linear  SDVF  based  on  a 
natural  scale  expressed  in  months,  with  breakpoints  at  12  and  36  months.  Due  to 
JIEDDO's  strong  preference  for  proposals  that  can  be  fielded  within  one  year  (JIEDDO, 
2007),  60%  of  the  value  is  awarded  to  proposals  that  meet  this  criterion.  In  addition,  the 
return  to  scale  decreases  after  36  months  because  JIEDDO  places  a  very  low  priority  on 
these  long  term  development  efforts.  In  order  to  obtain  the  input  for  the  Fielding 
Timeline  SDVF,  an  estimated  date  for  delivery  of  first  article  could  be  solicited  from  the 
offeror  during  the  BIDS  process. 


Figure  14.  Fielding  Timeline  SDVF 
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Operations  Burden  is  a  decreasing  exponential  SDVF  using  a  natural  scale 
expressed  in  percent  of  maximum  capacity  of  the  equipment  or  environment  in  which  the 
system  will  be  employed.  A  proposed  system  must  be  at  or  below  20%  of  maximum  host 
system  capacity  to  get  half  the  available  value,  because  most  host  systems— whether  they 
are  humans,  vehicles,  or  communications  channels— are  already  operating  near,  at  or  over 
their  maximum  capacity.  The  JIEDDO  ITB  could  compare  system  data  provided  by  the 
proponent  with  the  known  capacity  of  the  host  environment.  The  SDVF  exponential  for 
Operations  Burden  is  defined  by: 

loo-x 

Value  (x)  =  — —jog—  (3) 

l-e"22“ 

/  \ 


%  of  Maximum  Capacity 

v _ y 

Figure  15.  Operations  Burden  SDVF 

Work  Load  is  a  decreasing  categorical  SDVF  derived  from  an  exponential  curve. 
It  uses  a  natural  scale  expressed  in  interaction  minutes  per  hour  in  increments  of  0- 1 
minute,  2-5  minutes,  6-15  minutes,  16-30  minutes,  and  >  30  minutes.  A  proposed  system 
must  require  only  5  minutes  of  user  interaction  hourly  to  get  a  value  score  of  0.75.  End 
users  in  mission  areas  affected  by  the  IED  threat  are  already  task  saturated,  and  C-IED  is 
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just  one  of  the  things  they  are  responsible  for.  It  is  critical  that  C-IED  systems  not  detract 
from  their  users'  awareness  of  and  attention  to  the  surrounding  combat  situation. 

The  underlying  exponential  for  the  Work  Load  SDVF  is  defined  by: 

40— X 

Value  (x)  =  1  e  7 (4) 

l-e^28 


Figure  16.  Work  Load  SDVF 


Training  Time  is  another  decreasing  exponential  SDVF  using  a  natural  scale 
expressed  in  hours.  An  operations  SME  could  determine  the  estimated  training  time 
required  by  a  proposed  system.  Since  deploying  troops  already  require  extensive  training 
before  deploying  to  theater,  a  system  must  take  eight  or  fewer  hours  of  training  time  to 
get  50%  of  the  available  value.  The  SDVF  exponential  for  Training  Time  is  defined  by: 

40— X 

Value(x )  =  1  e  (5) 

l-eI^2 
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Figure  17.  Training  Time  SDVF 


Training  Program  Maturity  is  a  categorical  increasing  SDVF  based  on  a 
constructed  Training  Readiness  Level  (TRNL)  scale  developed  by  JIEDDO  and  located 
at  Appendix  A.  The  TRNL  of  a  system  could  be  assessed  by  the  JIEDDO  ITB.  There  is 
a  return  to  scale  from  0.15  to  0.6  when  moving  from  TRNL-2  to  TRNL-3  due  to 
JIEDDO ’s  requirement  that  a  proposal  be  at  TRNL-3  before  transitioning  into  the 
initiative  evaluation  stage. 


Figure  18.  Training  Program  Maturity  SDVF 


45 


Additive  Value  Function 


After  all  SDVF  inputs  are  detennined  and  values  calculated  for  each  evaluation 
measure,  the  following  additive  value  function  will  be  used  to  calculate  the  overall  value 
of  each  proposal: 

V(X)  -056vTenets(Xi)  F  •176vQapS(Xi)  H"  .056vciass(Xi)  .  1 1 2vTimeToCounter(Xj)  4~ 

■  1  lVjechPerj(Xi)  ,056vsujt(Xi)  4"  ,091  Vlnterop(Xi)  . 037vjechRisk(Xi )  -056vFieldTime(Xi)  "t" 

■  087vOpsBurden(Xi)  •  l  Viyorl,l,oacl(Xi)  "F  ■05VTrngTime(Xi)  -01 3VTrngMaturity(Xi)  (6) 

where 

X  =  the  proposal  under  consideration 

V(X)  =  the  overall  value  score  of  proposal  X 

x;  =  the  raw  data  input  associated  with  proposal  attribute  i 

Vj(xi)  =  the  score  of  proposal  attribute  i  against  its  associated  evaluation  measure  j. 
The  complete  hierarchy  with  global  weights  and  evaluation  measures  is  shown  in  Figure 
19.  The  complete  decision  model,  to  include  value  hierarchy,  weights,  SDVFs  and 
evaluation  measures,  was  then  validated  with  JIEDDO  and  appropriate  adjustments  were 
made. 
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Figure  19.  Complete  Value  Hierarchy 


Model  Verification 

Verification  of  the  completed  decision  model  was  accomplished  using  proposals 
that  have  been  previously  evaluated  within  the  BIDS  process.  The  intent  of  the 
verification  process  was  to  assess  the  results  of  decision  model  in  comparison  to  the 
current  process  and  determine  what,  if  any,  differences  might  exist.  In  fact,  slight 
differences  were  expected  since  the  decision  model,  with  its  increased  number  of 
evaluation  measures,  more  thoroughly  assesses  the  merits  of  a  submitted  proposal. 

In  addition  to  simply  assessing  the  scores  and  rank  order  of  evaluated  proposals, 
application  of  the  decision  model  with  proposals  that  have  already  been  assessed  within 
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the  BIDS  process  provided  significant  insight  into  the  current  evaluation  process  as  well 
as  highlighting  requirements  for  properly  implementing  the  decision  model. 

Selection  of  Proposals  for  Evaluation 

The  selection  of  proposals  for  use  in  verification  of  the  decision  model  was 
intended  to  capture  a  cross  section  of  proposals  that  have  been  recently  evaluated  within 
the  BIDS  process.  In  general  there  are  three  categories  of  proposal  evaluation 
recommendations  that  emerge  from  the  BIDS  process:  accept,  reject  and  express  reject. 
Proposals  can  be  express  rejected  due  to  improper  format  or  inadequate  information 
included  within  the  submission  packet  or  simply  due  to  obvious  lack  of  technical  merit. 
These  proposals  were  not  used  in  the  validation  to  prevent  problems  with  inadequate 
evaluation  material. 

Thirty  proposals  were  selected  for  use  in  the  verification  process  with  the  intent  of 
having  an  approximately  even  number  that  were  accepted  and  rejected.  (Due  to 
classification  issues  associated  with  the  contribution  of  prioritized  capability  gaps  to 
individual  proposal  scores,  all  proposals  are  referred  to  by  letters  only.  The  classified  key 
linking  the  proposals  to  their  letter  codes  can  be  obtained  by  contacting  the  AFIT 
Systems  Engineering  Department,  Dr.  John  Colombi.)  The  final  breakdown  included 
thirteen  accepted  proposals,  all  that  were  available  on  the  BIDS  portal  at  the  time  of  data 
collection,  and  seventeen  rejected  proposals  covering  all  areas  of  submission  for  BAA 
07-01.  Although  the  intent  was  to  cover  both  BAA  07-01  and  BAA  08-01  in  the 
evaluation,  the  only  proposals  that  were  available  for  BAA  08-01  were  express  rejected 
and  therefore  not  used. 
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Scoring 

In  order  to  validate  the  decision  model  while  staying  within  the  timeline  of  the 
research  project,  scoring  of  the  individual  evaluation  measures  for  each  proposal  was 
conducted  by  the  research  team.  Scores  for  measures  that  could  be  objectively 
determined  from  the  included  proposal  information  were  accomplished  first  by  the  entire 
team  by  dividing  the  proposals  among  the  team  members.  Once  this  was  completed,  each 
member  of  the  research  team  was  assigned  two  or  three  evaluation  measures,  based  on 
background  if  applicable,  and  the  evaluation  measures  assigned  were  scored  for  all  thirty 
proposals.  This  was  accomplished  to  ensure  consistency  in  scoring  and  allow  the  team 
members  to  establish  some  background  for  their  assigned  evaluation  measures.  The  final 
proposal  scoring  can  be  found  in  Appendix  B.  Once  complete,  the  scores  were  entered 
into  the  Hierarchy  Builder  software  to  calculate  and  provide  a  visual  representation  of  the 
value  scores  for  the  proposals,  and  allow  for  analysis  of  the  decision  model  results. 

Figure  20  illustrates  the  contribution  of  individual  values  to  the  total  score  for  each  of  the 
30  proposals. 
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□  Technology  Readiness  Level 


Figure  20.  Proposals  in  Order  of  Overall  Value  Score 


Sensitivity  Analysis 

Of  particular  importance  to  the  evaluation  of  the  decision  model  performance  is 
the  assessment  of  the  sensitivity  of  each  value  to  changes  in  the  weighting.  This  analysis 
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provides  insight  to  the  impact  of  changes  in  weighting  and  can  point  out  the  need  for 
further  investigation  as  to  the  weighting  of  particular  values  within  the  model.  Due  to  the 
large  number  of  proposals  that  were  scored  in  the  model,  sensitivity  analysis  was 
conducted  on  a  reduced  set  of  ten  proposals  on  the  expected  boundary  of  accepted  and 
rejected  proposals.  Evaluation  of  the  sensitivity  of  weighting  for  proposals  that  fell  in 
this  area  was  intended  to  provide  insight  into  whether  weighting  changes  would 
potentially  adjust  scores  to  more  consistently  match  the  results  of  the  current  BIDS 
process. 

The  primary  sensitivity  analysis  effort  was  conducted  for  each  value  by 
considering  a  swing  of  only  that  value’s  weight  from  zero  to  one.  If  the  preferred 
proposal  changed  as  weights  were  moved  in  modest  increments  from  the  current  model 
levels  then  the  value  was  considered  to  be  sensitive  to  weight  changes.  It  was  expected 
that  measures  for  which  all  proposals  scored  consistently  would  show  little  or  no 
sensitivity  to  weight  changes  while  those  with  widely  varying  scores  could  be  sensitive  to 
both  increases  and  decreases  in  weighting  changes.  In  addition,  some  values  would 
likely  show  sensitivity  in  only  one  direction  of  changes  in  weighting.  This  infonnation 
could  be  used  for  further  investigation  with  the  decision  maker  to  detennine  if  more 
detailed  evaluation  of  weighting  of  values  was  justified.  The  sensitivity  charts  for  all 
individual  evaluation  measures  are  included  in  Appendix  C. 

In  addition  to  the  above  described  “one  way”  sensitivity  analysis,  the  research 
team  conducted  “two  way”  sensitivity  analysis  on  the  second  tier  of  the  hierarchy  to 
determine  if  the  method  was  feasible  for  this  project  and  if  further  insight  into  weighting 
sensitivity  could  be  obtained.  This  analysis  was  conducted  by  varying  the  weight  of  two 
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of  the  values  on  that  tier  at  the  same  time  while  proportionally  allocating  the  remaining 
weight  balance  to  the  rest  of  the  values.  Since  this  was  conducted  on  tier  two  with  only 
three  values,  the  third  value  received  the  entire  balance  of  weight  that  was  not  assigned  to 
the  two  variable  value  weights.  The  results  were  then  plotted  to  show  regions  where 
different  proposals  would  be  selected  as  the  most  preferred  and  highlight  the  boundaries 
between  these  areas  where  weighting  is  sensitive  to  changes.  Due  to  the  continuous 
arrival  process  of  the  proposal  set,  the  use  of  sensitivity  analysis  to  alter  weights  is  likely 
to  take  a  back  seat  to  the  use  of  results  to  support  selection  decisions  of  closely  scored 
proposals.  The  two  way  sensitivity  helps  this  effort  by  highlighting  the  proposals  within 
a  selected  set  that  would  be  preferred  in  all  weighting  scenarios. 
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IV.  Results  and  Analysis 


Overview 

This  chapter  covers  the  results  of  the  application  of  the  IED  Defeat  Hierarchy 
discussed  in  Chapter  3  to  30  C-IED  proposals,  and  the  associated  sensitivity  analysis. 

The  hierarchical  model  determined  a  rank  ordered  list  of  30  alternatives,  shown  in  Table 
13,  resulting  from  the  scoring  process.  Sensitivity  analysis  on  the  weights  determined  for 
the  hierarchy  was  conducted  to  allow  further  insight  into  the  validity  of  the  model,  and  to 
reveal  where  further  examination  may  be  needed  to  more  accurately  represent  the 
decision  maker’s  values. 


Tab! 

e  13.  Rank  Ordered  Alternatives 

Proposal 

Score 

Proposal 

Score 

Proposal 
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.446 

CC 
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A* 
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.539 

K* 

.387 

J* 

.599 

D* 

.539 

V 

.366 

B* 

.585 

c* 

.528 

M* 

.363 

X 

.568 

L* 

.494 

H* 

.168 

Evaluation  Assumptions  and  Results 

The  information  available  from  the  JIEDDO  BIDS  Web  portal  was  used  for  the 
evaluation  of  the  30  alternatives.  White  papers  and  quad  charts  describing  the  technical 
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aspects,  system  features,  risk,  cost  and  schedule  as  approximated  by  the  contractor  were 
obtained  through  the  web  site.  Evaluations  completed  by  JIEDDO  subject  matter  experts 
were  also  available  for  each  proposal.  While  many  of  the  evaluation  measures  could  be 
readily  scored  from  information  contained  in  the  white  papers  and  quad  charts,  others 
were  more  subjective  and  certain  assumptions  had  to  be  made  to  ensure  consistent 
scoring  across  the  set  of  30  proposals. 

Research  and  Development  Proposals 

There  were  five  C-IED  proposals  (B,  E,  F,  J  and  DD)  that  were  pure  R&D  efforts. 
In  other  words,  their  deliverable  was  a  research  paper  instead  of  a  prototype  system. 
Scoring  these  proposals  was  a  challenge  because  many  of  the  hierarchy’s  values  depend 
on  assessing  a  fieldable  product.  The  team  ensured  R&D  proposals  were  consistently 
scored  by  assigning  them  the  same  inputs  for  the  evaluation  measures  of  Time  to  Counter 
(60  months),  Technical  Perfonnance  (1),  Suitability  (5),  Interoperability  (None), 
Operations  Burden  (0%),  Work  Load  (0-1  minutes  per  hour),  and  Training  Time 
(0  hours).  These  input  figures  were  selected  because  they  most  closely  matched  the  value 
definitions  as  they  apply  to  a  pure  R&D  proposal.  The  actual  figures  would  clearly  be 
different  for  a  C-IED  system  or  systems  developed  as  a  result  of  the  funded  R&D  effort. 
Gap  Impact 

The  value  of  Gap  Impact  was  assessed  using  JIEDDO ’s  May  2007  list  of 
capability  gaps.  Due  to  the  broad  scope  of  one  of  the  gaps  (“Improve  standoff  detection 
of  vehicle  borne  IEDs,  IEDs,  suicide  bombers  and  maritime  IEDs,”)  several  diverse 
proposals  addressed  that  gap  and  as  a  result  received  the  same  score  for  Gap  Impact. 
Conversely,  other  gaps  such  as  “Standoff  neutralization  that  preserves  forensic  integrity” 
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were  more  narrowly  defined.  As  a  result,  although  many  proposals  in  the  study  sample 
fell  under  the  tenet  of  Neutralize,  only  one  met  the  requirement  to  preserve  forensic 
integrity.  The  remaining  Neutralize  proposals  did  not  address  any  gap  and  were  awarded 
a  score  of  zero  for  this  value. 

Comparison  of  the  values  of  Gap  Impact  (light  blue)  and  Technical  Performance 
(cream)  in  the  proposal  rack  and  stack  at  Figure  20  reveals  many  proposals  that  scored 
well  in  performance  but  did  not  rise  high  in  the  ranking  because  they  did  not  address  a 
high  priority  gap  or  did  not  address  a  gap  at  all.  This  amply  demonstrates  the  decision 
maker’s  preference  for  systems  that  are  relevant  to  the  current  operational  needs  of  the 
warfighter  over  systems  that  perfonn  well  but  do  not  target  a  gap.  It  also  highlights  the 
need  for  JIEDDO  to  maintain  a  current,  prioritized  gap  list. 

Classification 

Classification  was  measured  at  the  level  of  classification  of  the  white  paper  and 
quad  chart  of  the  proposal.  As  a  result,  all  the  proposals  but  two  were  assessed  as  For 
Official  Use  Only,  and  the  remaining  two  were  SECRET/REL.  Many  proposals  in  their 
final  fielded  form  will  have  a  higher  classification  that  would  be  a  truer  representation  of 
this  value,  but  the  research  team  was  not  able  to  determine  the  future  classification  as  this 
point.  This  measure  is  important  in  that  it  represents  how  easily  the  proposed  system  will 
be  proliferated  among  the  US  Armed  forces,  Coalition  partners,  and  Host  Nation  forces 
during  transition. 

Technical  Risk 

Similar  to  Classification,  Technical  Readiness  Level  (the  evaluation  measure  for 
Technical  Risk)  was  assessed  at  the  current  level  instead  of  the  expected  TRL  of  the  final 
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product  that  was  advertised  in  some  proposals.  However,  in  this  case,  the  current  TRL  as 
of  the  date  of  submission  is  the  appropriate  place  to  perfonn  an  assessment,  because  it 
will  impact  the  likelihood  of  a  proposal’s  development  into  a  viable  fielded  system. 

Time  to  Counter 

One  of  the  more  difficult  areas  for  scoring  was  Time  to  Counter.  This  evaluation 
was  intended  to  estimate  the  time  a  particular  system  would  operate  in  the  IED  Defeat 
effort  before  the  enemy  developed  tactics  to  counter  the  system.  Proposals  that  are  not 
directly  designed  to  detect  or  counter  IED  devices  generally  score  high  in  this  evaluation. 
As  an  example,  a  network  analysis  software  system  will  likely  not  be  countered  by  the 
enemy  as  its  primary  function  is  data  analysis.  On  the  other  hand,  proposals  that  are 
intended  to  detect  and  or  counter  IED  devices  were  scored  lower  based  on  an  estimate  of 
how  long  it  would  take  the  enemy  to  detennine  how  their  devices  are  being  affected  by 
the  proposed  system.  If  the  enemy  counter  tactic  was  to  stop  using  the  targeted  IED 
device  then  the  proposed  system  was  considered  to  have  continuing  effectiveness. 
However,  if  the  enemy  counter  tactic  allowed  the  continued  use  of  the  IED  device  with 
modifications  that  significantly  reduced  or  negated  the  effectiveness  of  the  proposed 
system  it  was  considered  to  be  countered.  Although  an  in  depth  analysis  was  not 
conducted  for  each  proposal  considering  trends  of  enemy  counter  tactics  development, 
the  Time  to  Counter  score  was  estimated  by  a  general  assessment  of  how  difficult  it 
would  be  to  detennine  how  the  IED  devices  were  being  targeted  and  how  difficult  a 
counter  tactic  would  be  to  develop. 
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Operations  Burden 

Operations  Burden  was  a  complicated  field  for  scoring  since  it  had  to  be 
measured  in  terms  of  percent  of  maximum  capacity  of  such  a  wide  variety  of  platfonns. 
Some,  such  as  soldier  or  vehicle  capacity,  were  easily  calculated.  Other  solutions,  like 
software  upgrades,  were  more  difficult  to  score.  Assumptions  were  made  about  the 
numerical  values  of  available  bandwidth  and  system  capacity  to  allow  the  scoring  of 
these  solutions.  Although  the  range  of  the  evaluation  measure  was  from  0%  to  100%, 
with  one  exception  the  highest  figure  assessed  for  this  value  was  20%.  (The  exception 
was  a  hardened  shelter  system  that  is  not  designed  to  be  transported  once  it  is  in  place.) 
This  indicates  there  may  be  some  benefit  to  readjusting  the  maximum  range  of  the  SDVF 
to  reduce  the  inactive  portion  of  the  independent  axis. 

Work  Load 

The  majority  of  systems  that  received  low  scores  for  this  value  (>  30  minutes  per 
hour  of  required  interaction)  were  sensor  systems  that  required  a  human  operator  to 
monitor  some  type  of  display  for  an  indication  of  a  threat.  Higher  scores  were  awarded 
to  those  systems  that  had  automatic  monitoring  and  could  audibly  cue  the  human  operator 
when  their  attention  was  required. 

Technical  Performance 

Scores  for  Technical  Performance  fell  almost  exclusively  in  the  1  to  3  range  out 
of  5.  Only  two  4s  and  no  5s  were  awarded.  There  was  a  wide  range  of  expected 
performance  within  each  of  the  two  groups  of  proposals  that  were  awarded  scores  of  2  or 
3.  This  indicates  that  the  assessment  scale  for  Technical  Performance  could  be  redefined 
or  expanded  to  provide  greater  distinction  between  proposals. 
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Other  Values 


Fielding  Timeline,  Interoperability,  Suitability  and  Training  Time  were  easily 
measured  in  most  cases  from  information  provided  in  the  white  papers,  quad  charts  and 
evaluator  comments.  Since  Training  Level  could  not  be  assessed  with  the  available 
information,  every  proposal  received  a  3. 

Overall  Scoring  Results 

The  overall  rack  and  stack  produced  by  scoring  the  selected  proposals  against  the 
decision  model  was  provided  in  Figure  20.  Proposals  that  were  rejected  by  the  evaluation 
conducted  in  the  BIDS  process  are  indicated  by  an  asterisk.  In  general,  the  priority 
reflected  previous  BIDS  decisions  to  either  accept  or  reject  a  proposal,  with  a  few 
exceptions. 

R&D  proposals  scored  artificially  high  in  the  decision  model’s  rack  and  stack 
(DD,  F,  E,  J,  and  B).  As  previously  discussed,  scoring  was  conducted  not  against  the 
system  which  might  be  developed  as  a  result  of  the  R&D,  but  on  the  actual  R&D 
deliverable  of  a  research  paper.  The  scoring  methodology  resulted  in  overly  optimistic 
scores  for  many  of  the  evaluation  areas  that  drove  the  R&D  proposals  almost  to  the  top  of 
the  rack  and  stack.  This  scoring  discrepancy  demonstrates  why  R&D  proposals  do  not  fit 
well  into  the  decision  model.  Figure  2 1  contains  a  rank  ordered  list  of  the  non-R&D 
proposals  that  were  scored  in  the  model. 
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Figure  21.  Non-R&D  Proposals  in  Order  of  Overall  Value  Score 
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The  two  non-R&D  outliers  were  proposal  P,  which  was  rejected  by  the  BIDS 
process  but  was  preferred  over  two  selected  proposals  by  the  decision  model,  and 
proposal  V,  which  was  accepted  by  the  BIDS  process  but  ranked  28th  by  the  model. 

Initially  the  research  team  was  most  interested  in  the  BIDS  rejected  proposals  that 
scored  in  the  upper  half  of  the  group  as  opposed  to  accepted  proposals  that  scored  in  the 
lower  half  However,  due  to  the  ability  of  the  JCAAMP  process  to  further  evaluate  and 
eliminate  proposals  prior  to  full  funding  it  is  less  critical  to  ensure  that  some  borderline 
proposals  are  selected  for  further  investigation  as  it  is  to  not  reject  those  proposals  which 
have  potential  to  aid  in  IED  defeat  efforts.  Therefore,  the  most  important  outliers  to 
consider  for  further  investigation  would  be  those  that  fell  into  the  rejected  region  but 
could  actually  provide  positive  results. 

Proposal  P  was  ranked  15th  by  the  decision  model.  It  gained  most  of  its  value 
from  Time  to  Counter,  because  it  is  a  “behind  the  scenes”  information  management 
system  that  would  not  be  easily  countered  by  the  enemy.  However,  it  scored  from 
mediocre  to  poor  in  other  areas,  and  the  sensitivity  analysis  discussion  that  follows  will 
demonstrate  that  no  combination  of  weights  would  make  proposal  P  the  preferred 
alternative. 

The  other  outlier,  proposal  V,  received  an  average  score  (3)  for  Technical 
Performance,  but  did  not  do  well  in  other  areas,  particularly  in  Gap  Impact.  Proposal  V 
does  not  address  a  gap  so  it  was  awarded  the  lowest  score.  Since  Gap  Impact  is  the  most 
heavily  weighted  value,  its  overall  ranking  was  significantly  lower  than  it  would  have 
been  if  it  had  targeted  a  gap. 
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Sensitivity  Analysis 

Sensitivity  analysis  was  conducted  on  a  reduced  set  of  ten  proposals  (X,  R,  T,  Y, 

S,  P*,  W,  U,  D*,  C*)  from  the  thirty  used  to  evaluate  the  decision  model.  This  group  of 
ten  was  selected  for  two  reasons.  First,  they  include  the  transition  point  from 
predominantly  BIDS  accepted  proposals  to  predominantly  BIDS  rejected  proposals. 
Second,  this  set  of  proposals  was  tightly  grouped  with  only  a  .04  value  difference  from 
the  highest  to  lowest  score. 

One  way  sensitivity  analysis  of  weights  was  conducted  to  see  how  the  preference 
of  the  proposal  set  changed  as  each  of  the  value  weights  was  transitioned  from  0  to  1 . 
Results  of  this  effort  can  be  seen  in  Appendix  C,  Sensitivity  Analysis.  The  graphs  show 
current  weighting  of  the  measure  and  total  value  for  each  of  the  ten  proposals  for  a  given 
range  of  weights.  The  sensitivity  results  can  be  broken  up  into  four  categories  of  weight 
changes:  sensitive  to  increases,  sensitive  to  decreases,  sensitive  to  both,  and  not 
sensitive. 

There  was  a  very  small  delta  between  the  highest  and  lowest  overall  score  for  the 
proposals,  so  it  was  expected  that  most  of  the  values  would  have  some  sensitivity  to 
changes  in  weights.  The  values  that  are  sensitive  to  increases  in  weight  values  include 
Needed  Capability,  Classification,  Time  to  Counter  and  Technical  Performance.  Those 
which  were  sensitive  to  decreases  in  weight  values  include  Operational  Performance, 
Tenets  Impacted,  Suitability,  Interoperability,  Technical  Risk,  Fielding  Timeline  and 
Training  Time.  Usability,  Gap  Impact  and  Work  Load  were  sensitive  to  both  increases 
and  decreases  in  weight  values.  Although  these  values  were  sensitive  to  changes  in 
weights,  the  overall  ranking  of  the  proposals  indicates  that  the  initial  weighting  allows  the 
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model  to  match  the  current  BIDS  process  fairly  well.  If  the  R&D  proposals  are  removed 
from  consideration,  this  model  selected  92%  of  the  proposals  accepted  by  BIDS  with 
only  one  rejected  outlier  in  the  top  12  proposals. 

The  lack  of  sensitivity  to  weight  changes  for  Operations  Burden  and  Training 
Program  Maturity  can  be  explained  by  the  scoring  of  the  proposals  for  these  measures. 
Training  Program  Maturity  could  not  be  evaluated  based  on  the  data  and  resulted  in  all 
proposals  being  scored  the  same.  Operations  Burden  was  also  difficult  to  score  and 
resulted  in  values  that  were  similar  for  most  of  the  proposals.  Although  lack  of 
sensitivity  usually  indicates  the  potential  to  eliminate  a  value  from  the  hierarchy,  in  this 
case  it  is  not  justified.  An  area  for  further  investigation  would  be  to  gather  high  fidelity 
data  for  these  two  measures  and  reassess  the  sensitivity  of  the  model  to  these  values. 

One  way  sensitivity  analysis  is  particularly  helpful  when  a  fixed  set  of  alternatives 
are  evaluated  since  changes  in  the  preference  order  of  the  alternative  set  can  be  observed. 
The  unique  situation  encountered  in  this  research  is  the  continuous  arrival  of  alternatives 
which  will  alter  the  sensitivity  with  each  newly  scored  alternative.  Since  the  evaluation 
of  the  model  was  conducted  with  a  discrete  set  of  alternatives,  the  sensitivity  analysis  is 
only  useful  for  initial  training  of  the  model.  Once  the  model  is  in  use  for  continuous 
evaluation  of  incoming  proposals  the  ability  to  adjust  the  weighting  based  on  sensitivity 
analysis  becomes  difficult.  One  area  for  further  investigation  as  to  the  performance  of 
this  model  with  respect  to  historic  BIDS  results  would  be  to  score  all  the  proposals  that 
have  been  received  to  date  as  a  fixed  data  set.  This  may  provide  more  insight  as  to  the 
sensitivity  of  the  weighting;  however,  there  would  be  a  fairly  significant  cost  in  time  and 
effort  for  this  to  be  accomplished. 
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In  addition  to  one  way  sensitivity  analysis,  the  research  team  explored  the 
usefulness  of  two  way  sensitivity  analysis  where  two  weights  are  moved  at  the  same  time 
and  all  others  are  adjusted  proportionately  to  their  current  value.  This  analysis  was 
conducted  on  the  second  tier  where  there  are  only  three  weighted  values— Needed 
Capability,  Operational  Performance  and  Usability.  Since  there  were  only  three  values, 
the  group  of  “others”  consisted  of  the  third  value,  and  in  essence  all  three  were  adjusted 
at  the  same  time.  The  results,  shown  in  Figure  22,  highlight  the  areas  where  one  of  the 
ten  proposals  would  be  the  most  preferred  if  weighting  were  adjusted  to  a  particular 
combination  of  Needed  Capability,  Operational  Performance,  and  Usability.  Each  comer 
of  the  figure  represents  where  one  of  the  three  values  has  a  weight  of  one  and  the  others 
have  a  weight  of  zero.  Needed  Capability  and  Usability  both  decrease  to  zero  as  they 
move  toward  the  Operational  Performance  corner,  while  Operational  Performance 
decreases  to  zero  as  it  moves  toward  the  hypotenuse  of  the  triangle.  The  white  circle 
indicates  where  the  current  weighting  values  he  and  provides  an  intuitive  presentation  of 
the  impact  of  changing  the  weights  at  this  level  in  the  hierarchy. 

The  basic  information  provided  by  this  analysis  does  not  differ  from  the  one  way 
analysis;  however,  it  does  provide  some  additional  insight.  Consider  the  weighting  tuple 
of  (Needed  Capability,  Operational  Perfonnance,  Usability).  If  the  weighting  tuple  is 
moved  to  increase  or  decrease  one  value  while  maintaining  a  proportionate  relationship 
between  the  remaining  values,  the  resulting  preferred  alternative  changes  are  the  same  as 
looking  at  the  one  way  sensitivity  graph.  However,  if  all  values  are  changed 
simultaneously  without  maintaining  any  relationship  to  the  original  weight  values,  this 
method  highlights  the  resulting  change  in  proposal  preference  not  captured  in  the  one 
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way  charts.  Finally,  the  fairly  high  sensitivity  indicated  by  the  one  way  analysis  is 
supported  by  the  location  of  the  current  weighting  represented  by  the  white  circle. 
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Note  1:  This  is  the  location  of  current  weighting  indicating  high  sensitivity  to  the 
weights  of  Needed  Capability,  Operational  Performance,  and  Usability  within  this  set  of 
proposals. 

Note  2:  Alternatives  D,  S  and  T  had  the  same  score  for  Needed  Capability  resulting  in 
all  three  being  preferred  in  this  extreme  case. 

Figure  22.  Two  Way  Sensitivity  Region  Plot  and  Proposal  List 


The  insight  provided  by  this  analysis  comes  from  its  ability  to  highlight  the 
proposals  that  would  be  most  preferred  for  any  weighting  combination  at  this  tier  of  the 
decision  model.  This  information  could  be  used  to  support  decisions  of  whether  to  accept 
or  reject  those  proposals  that  lie  on  or  near  acceptance  or  rejection  boundary.  This  figure 
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shows  six  proposals  (R,S,T,U,W,X)  that  would  be  preferred  with  only  minor  changes  to 
the  weighting  combination  which  could  be  used  to  support  selection  of  this  set  of  six  out 
of  the  ten  compared.  Of  note,  none  of  the  BIDS  rejected  proposals  are  highlighted  on  this 
figure. 
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V.  Conclusions  and  Recommendations 


The  focus  of  this  research  effort  was  to  develop  a  decision  model  for  C-IED 
proposal  selection,  not  to  make  change  recommendations  for  JIEDDO’s  selection 
process.  Following  the  accepted  process  of  developing  a  VFT  decision  model,  the 
student  team  initially  concentrated  on  JIEDDO’s  values  for  IED  defeat  and  the  evaluation 
measures  that  would  best  capture  those  values.  Only  after  the  evaluation  measures  were 
selected  was  the  JCAAMP  process  reviewed  to  determine  what  was  currently  being 
measured  and  what  changes  would  be  needed  to  effectively  implement  the  model. 

Throughout  the  processes  of  developing  the  decision  model,  reviewing  previous 
evaluations  of  the  sample  set  of  proposals  and  scoring  the  same  set  against  the  newly 
developed  model,  the  research  team  developed  several  recommendations  for  JIEDDO’s 
consideration. 

Prioritization  of  Capability  Gaps 

One  of  the  most  critical  supporting  processes  for  the  decision  model  is  the 
management  of  capability  gaps.  Since  the  Gap  Impact  value  is  the  most  heavily 
weighted,  scores  for  this  evaluation  measure  have  the  most  effect  on  a  proposal’s  overall 
score.  The  current  portfolio  of  systems  and  ongoing  evaluations  reference  a  variety  of 
capability  gaps  and  requirements  sources  including:  Joint  Urgent  Operational  Need 
Statements  (JUONs),  Multinational  Corps-Iraq  (MNC-I),  emails  from  units  in  theater,  as 
well  as  the  requirements  cited  from  the  BAA.  Given  the  number  of  C-IED  requirements 
on  record,  it  is  very  easy  to  tie  any  type  of  C-IED  proposal  to  a  gap.  However,  since 
these  requirements  are  coming  from  many  different  sources,  there  is  no  way  to  detennine 
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the  relative  importance  of  multiple  targeted  gaps.  This  is  where  a  comprehensive, 
regularly  updated  capability  gap  portfolio  will  reap  the  most  benefits.  The  gaps  should 
then  be  incorporated  into  the  Requirements  section  of  the  BAA  to  ensure  offerors  clearly 
understand  JIEDDO’s  priorities  at  the  time  of  submission. 

An  additional  benefit  of  a  robust  gap  management  process  is  that  it  will  address 
the  issue  of  technology  duplication  addressed  in  the  current  BIDS  process.  As  multiple 
C-IED  systems  are  fielded  against  a  particular  need,  the  associated  gap  will  be  demoted 
on  or  removed  from  the  master  gap  listing. 

Lastly,  gaps  should  be  defined  in  terms  of  similar  breadth  to  the  maximum  extent 
possible.  For  example,  “standoff  detection  of  IEDs”  is  much  broader  than  “backtrack 
vehicles  in  a  rural  environment.”  This  leads  to  many  of  the  proposals  being  binned  into 
one  gap  and  receiving  the  full  benefit  of  that  gap’s  priority  score,  when  they  are  really 
addressing  different  pieces  of  the  gap. 

Evaluation  Process  Recommendations 
Time  to  Counter 

Currently,  JIEDDO’s  Technical  Gaming  Team  performs  an  exhaustive 
assessment  of  expected  “red”  (enemy)  and  “blue”  (friendly)  actions  and  reactions 
regarding  specific  C-IED  systems.  However,  this  assessment  is  not  performed  until  after 
a  proposal  is  selected  through  BIDS  and  enters  the  Equip  phase  (recall  Figure  2)  of  the 
JCAAMP  process.  The  research  team  recommends  that  the  TGT  support  the  BIDS 
process  by  acting  as  the  subject  matter  experts  for  assessing  Time  to  Counter  by 
performing  a  preliminary,  abbreviated  assessment  of  enemy  activities  that  impact  each 
C-IED  proposal  under  consideration. 
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Classification 

Although  the  classification  of  a  proposal  impacts  its  overall  usefulness,  most 
proposals  are  at  a  lower  classification  level  during  the  BIDS  process  than  they  will  be 
when  fielded.  Therefore,  the  score  for  this  evaluation  measure  will  not  return  an  accurate 
measure  of  a  proposal’s  value  related  to  classification.  The  JIEDDO  ITB  could  resolve 
this  discrepancy  by  assessing  the  anticipated  classification  of  the  proposal’s  fielded 
system. 

Training  Program  Maturity 

The  BIDS  process  currently  does  not  require  offerors  to  submit  documentation  of 
a  training  program  as  part  of  their  proposal.  Therefore,  there  no  is  no  data  with  which  to 
score  proposals  against  the  value  of  Training  Program  Maturity.  To  address  this  issue, 
JIEDDO  should  consider  requiring  training  program  documentation  as  part  of  the  white 
paper. 

Evaluator  Assignments 

The  research  team’s  review  of  previous  evaluator  comments  for  the  30  selected 
proposals  revealed  several  inconsistencies  in  the  scoring  process.  Currently,  an  evaluator 
scores  all  measures  of  a  proposal,  not  just  ones  on  which  they  have  subject  matter 
expertise.  For  instance,  many  evaluators  input  the  same  comments  into  their  responses 
for  several  evaluation  measures,  or  stated  that  they  were  not  qualified  to  respond  in  a 
particular  area.  Different  evaluator  comments  also  seemed  to  indicate  varying 
interpretations  of  the  evaluation  measures,  e.g.  one  evaluator  would  comment  on 
technology  risk  under  the  “Technology”  measure  while  another  would  comment  on  it 
under  the  “Schedule  Risk”  measure. 
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In  order  to  resolve  these  discrepancies,  it  is  recommended  that  the  JIEDDO  ITB 
assign  specific  evaluation  measures  to  individual  evaluators  according  to  their  area  of 
expertise,  as  described  in  Chapter  3.  For  instance,  some  measures  would  require  a 
technical  expert,  while  others  would  require  a  C-IED  operations  expert  or  a  Red  Team 
analyst.  This  would  preclude  evaluators  from  being  asked  to  assess  issues  outside  their 
area  of  expertise. 

Evaluator  Training 

It  is  further  recommended  that  the  value  definitions  and  evaluation  instructions 
contained  in  Chapter  3  be  provided  to  the  evaluators.  Proper  and  thorough  evaluator 
training  is  the  cornerstone  of  successfully  using  the  value  model.  Evaluators  do  not  need 
to  be  familiar  with  the  single  dimensional  value  function  underlying  their  assigned 
evaluation  measure — they  just  need  to  know  the  value  definition  and  associated 
instructions  and  background  material.  In  fact,  knowing  the  fonn  of  a  value  function  may 
actually  make  it  more  difficult  to  make  an  objective  assessment. 

A  good  example  of  the  importance  of  evaluator  training  is  the  most  heavily 
weighted  value— Gap  Impact.  Some  capability  gaps  are  narrowly  defined,  such  as 
“Standoff  neutralization  that  preserves  forensic  integrity.”  This  means  that  a  proposal 
must  not  only  be  able  to  neutralize  IEDs,  but  it  must  be  able  to  do  so  while  allowing  for 
forensic  analysis  of  the  detected  IED.  This  rules  out  predetonation  systems  and  radio 
frequency  suppression  systems  such  as  Counter  Radio  Controlled  Electronic  Warfare 
(CREW),  which  typically  neutralize  IEDs  without  detecting  them.  (Obviously,  forensic 
analysis  can  not  be  perfonned  on  an  undetected  IED.)  There  are  many  predetonation  and 
RF  suppression  C-IED  proposals  that  would  appear  to  address  the  standoff  neutralization 
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gap  quoted  above.  An  untrained  evaluator  could  easily  make  a  scoring  error  when 
assessing  a  batch  of  such  proposals,  leading  to  an  erroneous  ranking  of  the  overall 
portfolio  under  consideration. 

Application  to  Systems  Engineering 

Several  areas  of  AFIT’s  overall  research  effort— of  which  this  paper  is  a  small 
part— could  be  facilitated  by  the  application  of  system  engineering  concepts.  In 
particular,  an  “as-is”  architecture  could  have  been  developed  at  the  outset  of  a 
cooperative  research  effort  comprising  multiple  teams.  The  BIDS  process  that  would 
host  this  paper’s  decision  model  is  part  of  the  larger,  more  complex  JCAAMP  process.  A 
JIEDDO  architecture  would  have  framed  the  specific  C-IED  proposal  selection  decision 
problem,  as  well  as  the  other  areas  of  JIEDDO-sponsored  research,  in  a  useful  context. 

In  addition  to  benefiting  from  the  application  of  systems  engineering  tools  such  as 
architecture,  this  decision  model  would  also  benefit  the  larger  systems  engineering  effort 
of  developing  and  fielding  the  selected  C-IED  systems.  The  weights,  evaluation 
measures  and  single  dimensional  value  functions  developed  for  this  paper’s  value 
hierarchy  could  be  applied  during  the  requirements  development  phase  of  a  particular 
system.  For  example,  they  could  be  used  as  starting  points  for  the  preparation  of 
threshold  and  objective  performance  specifications  to  be  applied  during  operational 
testing. 

Areas  for  Further  Research 
Complexity  of  Technical  Risk 

After  scoring  the  30  sample  proposals  against  the  decision  model  in  conjunction 
with  reviewing  the  comments  of  previous  BIDS  evaluators,  the  research  team  determined 
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that  the  value  of  Technical  Risk  is  really  the  combination  of  two  related  values — 
technical  feasibility  and  technology  readiness.  Technical  feasibility  can  best  be  described 
as  the  answer  to  the  question  “What  is  the  likelihood  that  this  thing  will  work?” 
Technology  readiness,  usually  assessed  by  the  widely  used  Technology  Readiness  Level 
(TRL)  scale  in  Appendix  A,  answers  the  question  “To  what  fidelity  has  this  system  been 
proven?”  Although  a  TRL  can  be  (and  frequently  is)  used  to  assess  technical  feasibility, 
they  are  not  precisely  equivalent.  For  instance,  many  proposals  received  by  JIEDDO  are 
highly  speculative  with  little,  if  any,  technical  theory  to  back  up  their  assertions  of 
predicted  perfonnance.  These  systems  should  be  assessed  as  having  both  low  technical 
feasibility  and  low  technology  readiness.  By  contrast,  another  proposed  system  might  be 
based  on  the  integration  of  several  proven  technologies,  so  it  would  have  a  relatively  high 
technical  feasibility,  but  because  it  is  still  a  “paper-only”  idea  it  should  score  a  low  TRL. 
Handling  of  Non-Materiel  Solutions 

While  the  decision  model  is  flexible  enough  to  handle  different  classes  of  materiel 
C-IED  solutions  such  as  vehicle  mounted,  soldier-borne  or  software  solutions,  it  is  not  a 
good  fit  for  non-materiel  or  R&D  only  proposals.  Evaluation  measures  such  as 
Operations  Burden,  Interoperability,  Suitability,  Technical  Perfonnance  and  Technical 
Risk  are  not  effective  measures  for  non-materiel  proposals.  As  a  result,  proposals  that 
focus  on  important  theater  requirements  such  as  training  or  influence  operations  would 
not  score  well  in  the  model  even  if  they  have  significant  potential  for  successful 
implementation.  JIEDDO  should  evaluate  these  proposals  as  a  separate  group  using 
another  decision  model. 
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Investment  Analysis 

The  focus  of  this  paper  has  been  the  determination  of  the  value  of  C-IED 
proposals.  However,  as  the  proposals  continue  through  the  JCAAMP  process  they  will 
become  initiatives  upon  leaving  the  “Vet”  phase,  and  then  be  presented  to  JIEDDO 
leadership  for  a  funding  decision  upon  entering  the  “Equip”  phase  (recall  Figure  1).  In 
order  to  perform  an  investment  analysis  of  a  group  of  initiatives,  the  relative  cost-value 
ratio  of  each  initiative  must  be  considered  in  the  context  of  the  funding  constraints  in 
place  at  that  time. 

Conclusion 

Comparison  of  the  results  of  the  research  team’s  rack  and  stack  of  the  30  sample 
proposals  with  JIEDDO ’s  binary  accept  or  reject  process  reveals  a  positive  correlation 
between  the  two.  This  indicates  that  the  current  selection  process  reflects  the  values  of 
JIEDDO  decision  makers  at  a  macro  level. 

This  decision  model  will  provide  traceable,  repeatable  and  defensible  scoring  of 
IED  defeat  proposals  in  support  of  JIEDDO ’s  funding  decisions.  Effective 
implementation  of  the  model  will  rely  on  its  use  in  conjunction  with  the  other  process 
recommendations  in  this  thesis.  The  scoring  infonnation  will  provide  data  for  trend 
analysis  and  responses  to  external  inquiries  about  JIEDDO’ s  selection  decisions. 
Ultimately,  the  decision  model  will  provide  support  and  justification  for  JIEDDO ’s 
critical,  highly  visible  and  potentially  life-saving  decisions  regarding  acceptance  or 
rejection  of  IED  defeat  proposals. 
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Appendix  A:  JIEDDO  Assessment  Scales 


echnology  Readiness  Levels  (JIEDDO,  2007) 


TRL 

Description 

1 

Basic  principles  observed  and  reported.  Lowest  level  of  technology 
maturation.  At  this  level,  scientific  research  begins  to  be  translated  into 
applied  research  and  development.  Cost  to  achieve:  Very  low  “Unique” 
cost  (investment  cost  is  borne  by  scientific  research  programs). 

2 

Technology  concept  and/or  application  formulated.  Once  basic  physical 
principles  are  observed,  then  at  the  next  level  of  maturation,  practical 
applications  of  those  characteristics  can  be  “invented”  or  identified.  At 
this  level,  the  application  is  still  speculative:  there  is  not  experimental 
proof  or  detailed  analysis  to  support  the  conjecture.  Cost  to  achieve: 

Very  low  “Unique”  cost  (investment  cost  is  borne  by  scientific  research 
programs). 

3 

Analytical  and  experimental  critical  function  and/or  characteristic  proof- 
of-concept.  At  this  step  in  the  maturation  process,  active  Research  and 
Development  (R&D)  is  initiated.  This  must  include  both  analytical 
studies  to  set  the  technology  into  an  appropriate  context  and  laboratory- 
based  studies  to  physically  validate  that  the  analytical  predictions  are 
correct.  These  studies  and  experiments  should  constitute  proof-of- 
concept  validation  of  the  applications/concepts  formulated  at  TRL  2. 

Cost  to  achieve:  Low  “Unique”  cost  (technology  specific). 

4 

Component  validation  in  laboratory  environment.  Following  successful 
proof-of-concept  work,  basic  technological  elements  are  integrated  to 
establish  that  the  “pieces”  will  work  together  to  achieve  concept¬ 
enabling  levels  of  performance  for  a  component.  This  validation  must 
be  devised  to  support  the  concept  that  was  fonnulated  earlier,  and 
should  also  be  consistent  with  the  requirements  of  potential  system 
applications.  The  validation  is  relatively  “low-fidelity”  compared  to  the 
eventual  system:  it  could  be  composed  of  ad-hoc  discrete  components  in 
a  laboratory. 

5 

Component  validation  in  relevant  environment.  At  this,  the  fidelity  of 
the  component  being  tested  has  to  increase  significantly.  The  basic 
technological  elements  must  be  integrated  with  reasonably  realistic 
supporting  elements  so  that  the  total  applications  (component-level, 
sub-system  level  or  system-level)  can  be  tested  in  a  simulated  or 
somewhat  realistic  environment.  From  one-to-several  new  technologies 
might  be  involved  in  the  demonstration.  Cost  to  achieve:  Moderate 
“Unique”  cost  (investment  cost  will  be  technology  dependent,  but  likely 
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to  be  several  factors  greater  than  cost  to  achieve  TRL  4. 

6 

System/subsystem  model  or  prototype  demonstration  in  a  relevant 
environment.  A  major  step  in  the  level  of  fidelity  of  the  technology 
demonstration  follows  the  completion  of  TRL  5.  At  TRL  6,  a 
representative  model,  prototype  system  or  system  -would  be  tested  in  a 
relevant  environment.  Of  course,  the  demonstration  should  be 
successful  to  represent  a  true  TRL  6.  Not  all  technologies  will  undergo 
a  TRL  6  demonstration:  at  this  point  the  maturation  step  is  driven  more 
by  assuring  management  confidence  than  by  R&D  requirements.  The 
demonstration  might  represent  an  actual  system  application,  or  it  might 
only  be  similar  to  the  planned  application,  but  using  the  same 
technologies.  At  this  level,  several,  to  many  new  technologies  might  be 
integrated  into  the  demonstration.  Cost  to  achieve:  Technology  and 
demonstration  specific. 

7 

TRL  is  a  significant  step  beyond  TRL  6,  requiring  an  actual  system 
prototype  demonstration.  It  has  not  always  been  implemented  in  the 
past.  In  this  case,  the  prototype  should  be  near  or  at  the  scale  of  the 
planned  operational  system  and  the  demonstration  must  take  place  in  an 
operational  environment.  The  driving  purposes  for  achieving  this  level 
of  maturity  are  to  assure  system  engineering  and  development 
management  confidence  (more  than  for  purposes  of  technology  R&D). 
Therefore,  the  demonstration  must  be  of  a  prototype  of  that  application. 
Not  all  technologies  in  all  systems  will  go  to  this  level.  TRL  7  would 
normally  only  be  preformed  in  cases  where  the  technology  and  /or 
subsystem  application  Is  mission  critical  and  relatively  high  risk.  Cost 
to  achieve:  technology  and  demonstration  specific,  but  a  significant 
fraction  of  the  cost  of  TRL  8. 

8 

Actual  system  completed  through  test  and  demonstration.  Actual  system 
qualified  through  test  and  demonstration.  By  definition,  all 
technologies  being  applied  in  actual  systems  go  through  TRL  8.  In 
almost  all  cases,  this  level  is  the  end  of  true  ‘system  development’  for 
most  technology  elements.  This  might  include  integration  of  new 
technology  into  an  existing  system.  Cost  to  achieve:  Mission  specific; 
typically  highest  unique  cost  for  a  new  technology. 

9 

Actual  system  proven  through  successful  mission  operations.  By 
definition.  All  technologies  being  applied  in  actual  systems  go  through 
TRL  9.  In  almost  all  cases,  the  end  of  last  ‘bug  fixing’  aspects  of  true 
‘system  development’.  This  TRL  does  not  include  planned  product 
improvement  of  on  going  or  reusable  systems.  Cost  to  achieve:  Mission 
Specific;  less  than  cost  of  the  TRL  8. 
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raining  Levels  (JIEDDO,  2007) 


TRNL 

Description 

1 

Initial  -  Training  Program  does  not  exist  or. .  .Potential  training  program 
is  considered  but  lacks  any  formal  documentation.  Cost,  distribution, 
schedule,  functionality  and  maintenance  have  not  been  defined. 

2 

Repeatable  -  All  appropriate  items  in  TRL  1  are  satisfied.  Training 
program  is  defined  with  some  form  of  documentation.  Cost, 
distribution,  schedule,  functionality  and  maintenance  (all)  issues  have 
been  considered  to  the  point  of  initial  written  definition. 

3 

Defined  -  All  items  in  TRL  2  are  satisfied.  Long-term  management 
issues  (such  as  PM  and  FM)  have  been  considered.  Hand-off  to  future 
Service  sponsor  has  been  considered  and  initial  liaison  has  shown 
potential  concurrence  by  that  sponsor.  Prototype  hardware/software  has 
been  produced  and  demonstrated  in  an  equivalent  environment. 

Standards  for  production  have  been  defined. 

4 

Managed  -  All  items  in  TRL  3  are  satisfied.  Actual  hardware/software 
has  been  produced  and  demonstrated  in  an  actual  environment. 

Standards  for  production  have  been  achieved. 

5 

Optimizing  -  All  items  in  TRL  4  are  satisfied.  Processes  to  update  and 
improve  the  system  are  in  place  and  demonstrated.  System  is 
considered  ready  to  transition  to  a  Service. 
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Appendix  B:  Proposal  Scoring 
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Appendix  C:  Sensitivity  Graphs 
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